Skip to content

User Guide

This guide covers everything you need to start using gsql2rsql to transpile OpenCypher queries to Databricks SQL.

Looking for examples?

After learning the basics, check out complete query examples:


Installation

Bash
pip install gsql2rsql

Requirements:

  • Python 3.12+
  • Databricks Runtime 15+ only needed to execute the generated SQL

Try It Now

No database needed! Just define a schema and generate SQL:

Python
from gsql2rsql import GraphContext

# Table names are just strings - no database connection required
graph = GraphContext(
    nodes_table="my_nodes",
    edges_table="my_edges",
)
graph.set_types(
    node_types=["Person", "Company"],
    edge_types=["WORKS_AT"],
)

# Generate SQL from OpenCypher
sql = graph.transpile("""
    MATCH (p:Person)-[:WORKS_AT]->(c:Company)
    RETURN p.node_id, c.node_id
""")

print(sql)  # Copy this SQL to run on Databricks
Generated SQL Output
SQL
SELECT 
   _gsql2rsql_p_node_id AS node_id
  ,_gsql2rsql_c_node_id AS node_id
FROM (
  SELECT
     _left_0._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
    ,_left_0._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
    ,_left_0._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
    ,_right_0._gsql2rsql_c_node_id AS _gsql2rsql_c_node_id
  FROM (
    SELECT
       _left_1._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
      ,_right_1._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
      ,_right_1._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
    FROM (
      SELECT
         node_id AS _gsql2rsql_p_node_id
      FROM
        my_nodes
      WHERE (node_type = 'Person')
    ) AS _left_1
    INNER JOIN (
      SELECT
         src AS _gsql2rsql__anon1_src
        ,dst AS _gsql2rsql__anon1_dst
      FROM
        my_edges
      WHERE (relationship_type = 'WORKS_AT')
    ) AS _right_1 ON
      _left_1._gsql2rsql_p_node_id = _right_1._gsql2rsql__anon1_src
  ) AS _left_0
  INNER JOIN (
    SELECT
       node_id AS _gsql2rsql_c_node_id
    FROM
      my_nodes
    WHERE (node_type = 'Company')
  ) AS _right_0 ON
    _right_0._gsql2rsql_c_node_id = _left_0._gsql2rsql__anon1_dst
) AS _proj

GraphContext: Full Configuration

GraphContext is the recommended API for graph data stored in the Triple Store pattern (one nodes table + one edges table). It eliminates ~100 lines of schema boilerplate.

With Node/Edge Attributes

Python
from gsql2rsql import GraphContext

# Define schema with additional attributes
graph = GraphContext(
    nodes_table="catalog.schema.nodes",
    edges_table="catalog.schema.edges",
    extra_node_attrs={"name": str, "age": int, "score": float},
    extra_edge_attrs={"weight": float, "timestamp": str},
)

graph.set_types(
    node_types=["Person", "Company", "Account"],
    edge_types=["KNOWS", "WORKS_AT", "OWNS"],
)

# Now you can use those attributes in queries
sql = graph.transpile("""
    MATCH (p:Person)-[:WORKS_AT]->(c:Company)
    WHERE c.industry = 'Technology'
    RETURN p.name, c.name AS company
    LIMIT 100
""")

print(sql)
Generated SQL
SQL
SELECT 
   _gsql2rsql_p_name AS name
  ,_gsql2rsql_c_name AS company
FROM (
  SELECT
     _left_0._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
    ,_left_0._gsql2rsql_p_name AS _gsql2rsql_p_name
    ,_left_0._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
    ,_left_0._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
    ,_right_0._gsql2rsql_c_node_id AS _gsql2rsql_c_node_id
    ,_right_0._gsql2rsql_c_name AS _gsql2rsql_c_name
    ,_right_0._gsql2rsql_c_industry AS _gsql2rsql_c_industry
  FROM (
    SELECT
       _left_1._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
      ,_left_1._gsql2rsql_p_name AS _gsql2rsql_p_name
      ,_right_1._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
      ,_right_1._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
    FROM (
      SELECT
         node_id AS _gsql2rsql_p_node_id
        ,name AS _gsql2rsql_p_name
      FROM
        catalog.schema.nodes
      WHERE (node_type = 'Person')
    ) AS _left_1
    INNER JOIN (
      SELECT
         src AS _gsql2rsql__anon1_src
        ,dst AS _gsql2rsql__anon1_dst
      FROM
        catalog.schema.edges
      WHERE (relationship_type = 'WORKS_AT')
    ) AS _right_1 ON
      _left_1._gsql2rsql_p_node_id = _right_1._gsql2rsql__anon1_src
  ) AS _left_0
  INNER JOIN (
    SELECT
       node_id AS _gsql2rsql_c_node_id
      ,name AS _gsql2rsql_c_name
      ,industry AS _gsql2rsql_c_industry
    FROM
      catalog.schema.nodes
    WHERE (node_type = 'Company') AND ((industry) = ('Technology'))
  ) AS _right_0 ON
    _right_0._gsql2rsql_c_node_id = _left_0._gsql2rsql__anon1_dst
) AS _proj
LIMIT 100

GraphContext Parameters

Parameter Type Description
nodes_table str Full path to nodes table (e.g., catalog.schema.nodes)
edges_table str Full path to edges table
node_type_col str Column for node type (default: "node_type")
edge_type_col str Column for edge type (default: "relationship_type")
node_id_col str Column for node ID (default: "node_id")
edge_src_col str Column for edge source (default: "src")
edge_dst_col str Column for edge destination (default: "dst")
extra_node_attrs dict Additional node properties {name: type}
extra_edge_attrs dict Additional edge properties {name: type}
spark SparkSession Optional: for auto-discovery and execution
optimize_dead_tables bool Enable dead table elimination optimization (default: True)

Automatic System Properties

GraphContext automatically exposes system properties for use in queries, without needing to declare them in extra_node_attrs or extra_edge_attrs:

Node system properties:

Property Default Column Description
node_id node_id Unique node identifier
node_type node_type Node label/type

Edge system properties:

Property Default Column Description
src src Source node ID
dst dst Target node ID
relationship_type relationship_type Edge label/type

This allows queries like:

Python
# Filter by node_type without declaring it in extra_node_attrs
sql = graph.transpile("""
    MATCH (a)-[r]->(b)
    WHERE a.node_type IN ['Person', 'Company']
    RETURN a.node_id, r.relationship_type, b.node_id
""")

Expected Table Structure

Nodes table:

Column Description
node_id Unique node identifier
node_type Node label (Person, Company, etc.)
name, age, ... Node properties

Edges table:

Column Description
src Source node ID
dst Destination node ID
relationship_type Edge label (KNOWS, WORKS_AT, etc.)
weight, ... Edge properties

No-Label Nodes (Wildcard Matching)

GraphContext automatically enables no-label support, allowing nodes without explicit labels in queries:

Python
# Node 'a' has no label - matches ANY node type
sql = graph.transpile("""
    MATCH (a)-[:WORKS_AT]->(c:Company)
    RETURN a, c.name
""")

This is useful when:

  • You don't know or care about the source node type
  • You want to match multiple node types at once
  • You're exploring relationships without type constraints

Performance Impact

No-label nodes cause full table scans on the nodes table (no WHERE node_type = '...' filter). Use explicit labels whenever possible for production queries.


Variable-Length Paths

One of gsql2rsql's most powerful features is support for variable-length paths using WITH RECURSIVE.

Python
sql = graph.transpile("""
    MATCH path = (origin:Person {id: 12345})-[:KNOWS*1..3]->(friend:Person)
    RETURN friend.name, length(path) AS hops
    ORDER BY hops
""")

Key features:

  • Depth bounds: *1..3 means 1 to 3 hops
  • Cycle detection: Automatic ARRAY_CONTAINS checks prevent infinite loops
  • Path functions: length(path), nodes(path), relationships(path)
  • Path variable: Captures the entire path for analysis

Bidirectional BFS Optimization

When BOTH source AND target have equality filters on ID, bidirectional BFS can enable queries that would otherwise fail due to Spark's recursion limits.

Python
sql = graph.transpile(
    """
    MATCH path = (a:Person)-[:KNOWS*1..4]->(b:Person)
    WHERE a.node_id = 'alice' AND b.node_id = 'dave'
    RETURN nodes(path) AS path_nodes
    """,
    bidirectional_mode="recursive"  # Enable optimization
)
Generated SQL (bidirectional off vs recursive)
SQL
WITH RECURSIVE
  paths_1 AS (
    -- Base case: direct edges (depth = 1)
    SELECT
      e.src AS start_node,
      e.dst AS end_node,
      1 AS depth,
      ARRAY(e.src, e.dst) AS path,
      ARRAY(NAMED_STRUCT('src', e.src, 'dst', e.dst, 'relationship_type', e.relationship_type)) AS path_edges,
      ARRAY(e.src) AS visited
    FROM graph.edges e
    JOIN graph.nodes src ON src.node_id = e.src
    WHERE (relationship_type = 'KNOWS') AND (src.node_id) = ('alice')

    UNION ALL

    -- Recursive case: extend paths
    SELECT
      p.start_node,
      e.dst AS end_node,
      p.depth + 1 AS depth,
      CONCAT(p.path, ARRAY(e.dst)) AS path,
      ARRAY_APPEND(p.path_edges, NAMED_STRUCT('src', e.src, 'dst', e.dst, 'relationship_type', e.relationship_type)) AS path_edges,
      CONCAT(p.visited, ARRAY(e.src)) AS visited
    FROM paths_1 p
    JOIN graph.edges e
      ON p.end_node = e.src
    WHERE p.depth < 4
      AND NOT ARRAY_CONTAINS(p.visited, e.dst)
      AND (relationship_type = 'KNOWS')
  )
SELECT 
   _gsql2rsql_path_id AS path_nodes
FROM (
  SELECT
     sink.node_id AS _gsql2rsql_b_node_id
    ,sink.node_type AS _gsql2rsql_b_node_type
    ,source.node_id AS _gsql2rsql_a_node_id
    ,source.node_type AS _gsql2rsql_a_node_type
    ,p.start_node
    ,p.end_node
    ,p.depth
    ,p.path AS _gsql2rsql_path_id
    ,p.path_edges AS _gsql2rsql_path_edges
  FROM paths_1 p
  JOIN graph.nodes sink
    ON sink.node_id = p.end_node
  JOIN graph.nodes source
    ON source.node_id = p.start_node
  WHERE p.depth >= 1 AND p.depth <= 4 AND sink.node_type = 'Person' AND source.node_type = 'Person' AND (sink.node_id) = ('dave')
) AS _proj
SQL
WITH RECURSIVE
  forward_paths_1 AS (
    -- Depth 0: src node itself (for meeting with backward)
    SELECT
      src.node_id AS current_node,
      0 AS depth,
      ARRAY(src.node_id) AS path,
      CAST(ARRAY() AS ARRAY<STRUCT<src: STRING, dst: STRING>>) AS path_edges
    FROM graph.nodes src
    WHERE (src.node_id) = ('alice')

    UNION ALL

    -- Depth 1+: explore edges from src
    SELECT
      e.dst AS current_node,
      1 AS depth,
      ARRAY(e.src, e.dst) AS path,
      ARRAY(STRUCT(e.src AS src, e.dst AS dst)) AS path_edges
    FROM graph.edges e
    JOIN graph.nodes src ON src.node_id = e.src
    WHERE (relationship_type = 'KNOWS') AND (src.node_id) = ('alice')

    UNION ALL

    -- Recursive case: extend forward
    SELECT
      e.dst AS current_node,
      f.depth + 1 AS depth,
      CONCAT(f.path, ARRAY(e.dst)) AS path,
      CONCAT(f.path_edges, ARRAY(STRUCT(e.src AS src, e.dst AS dst))) AS path_edges
    FROM forward_paths_1 f
    JOIN graph.edges e
      ON f.current_node = e.src
    WHERE f.depth < 2
      AND NOT ARRAY_CONTAINS(f.path, e.dst)
      AND (relationship_type = 'KNOWS')
  ),

  backward_paths_1 AS (
    -- Depth 0: tgt node itself (for meeting with forward)
    SELECT
      tgt.node_id AS current_node,
      0 AS depth,
      ARRAY(tgt.node_id) AS path,
      CAST(ARRAY() AS ARRAY<STRUCT<src: STRING, dst: STRING>>) AS path_edges
    FROM graph.nodes tgt
    WHERE (tgt.node_id) = ('dave')

    UNION ALL

    -- Depth 1+: explore edges from tgt
    SELECT
      e.src AS current_node,
      1 AS depth,
      ARRAY(e.src, e.dst) AS path,
      ARRAY(STRUCT(e.src AS src, e.dst AS dst)) AS path_edges
    FROM graph.edges e
    JOIN graph.nodes tgt ON tgt.node_id = e.dst
    WHERE (relationship_type = 'KNOWS') AND (tgt.node_id) = ('dave')

    UNION ALL

    -- Recursive case: extend backward
    SELECT
      e.src AS current_node,
      b.depth + 1 AS depth,
      CONCAT(ARRAY(e.src), b.path) AS path,
      CONCAT(ARRAY(STRUCT(e.src AS src, e.dst AS dst)), b.path_edges) AS path_edges
    FROM backward_paths_1 b
    JOIN graph.edges e
      ON b.current_node = e.dst
    WHERE b.depth < 2
      AND NOT ARRAY_CONTAINS(b.path, e.src)
      AND (relationship_type = 'KNOWS')
  ),

  paths_1 AS (
    -- Intersection: paths that meet in the middle
    -- Use DISTINCT to deduplicate paths found via different meeting points
    SELECT DISTINCT
      f.path[0] AS start_node,
      b.path[SIZE(b.path) - 1] AS end_node,
      f.depth + b.depth AS depth,
      CONCAT(SLICE(f.path, 1, SIZE(f.path) - 1), b.path) AS path,
      CONCAT(f.path_edges, b.path_edges) AS path_edges
    FROM forward_paths_1 f
    JOIN backward_paths_1 b
      ON f.current_node = b.current_node
    WHERE f.depth + b.depth >= 1
      AND f.depth + b.depth <= 4
      AND SIZE(ARRAY_INTERSECT(SLICE(f.path, 1, SIZE(f.path) - 1), b.path)) = 0
  )
SELECT 
   _gsql2rsql_path_id AS path_nodes
FROM (
  SELECT
     sink.node_id AS _gsql2rsql_b_node_id
    ,sink.node_type AS _gsql2rsql_b_node_type
    ,source.node_id AS _gsql2rsql_a_node_id
    ,source.node_type AS _gsql2rsql_a_node_type
    ,p.start_node
    ,p.end_node
    ,p.depth
    ,p.path AS _gsql2rsql_path_id
    ,p.path_edges AS _gsql2rsql_path_edges
  FROM paths_1 p
  JOIN graph.nodes sink
    ON sink.node_id = p.end_node
  JOIN graph.nodes source
    ON source.node_id = p.start_node
  WHERE p.depth >= 1 AND p.depth <= 4 AND sink.node_type = 'Person' AND source.node_type = 'Person' AND (sink.node_id) = ('dave')
) AS _proj

Why use it:

  • Unidirectional BFS explores ALL paths from source (exponential growth)
  • Bidirectional BFS explores from both ends and prunes paths that don't meet
  • For small graphs: bidirectional has ~20-30% overhead
  • For large graphs: unidirectional fails (hits maxRowsPerIteration limit), bidirectional succeeds

Modes:

Mode Description
"off" Standard unidirectional BFS
"recursive" WITH RECURSIVE forward/backward CTEs (default)
"unrolling" Unrolled CTEs (best for depth ≤6)
"auto" Auto-select based on max_hops

Inline Filters

Use inline property filters for better performance:

Python
# Inline filter (optimized - pushed to source scan)
sql = graph.transpile("""
    MATCH (p:Person {status: 'active'})-[:KNOWS]->(friend)
    RETURN p.name, friend.name
""")

# Equivalent WHERE clause (less optimized)
sql = graph.transpile("""
    MATCH (p:Person)-[:KNOWS]->(friend)
    WHERE p.status = 'active'
    RETURN p.name, friend.name
""")

Inline filters are pushed down to the data source scan, reducing the amount of data processed.


Undirected Relationships

Use -[:REL]- (without arrow) for undirected relationships:

Python
sql = graph.transpile("""
    MATCH (a:Person)-[:KNOWS]-(b:Person)
    WHERE a.id = 123
    RETURN b.name
""")

This generates a UNION ALL query to match both directions efficiently.


Aggregations

Standard aggregation functions are supported:

Python
sql = graph.transpile("""
    MATCH (p:Person)-[:WORKS_AT]->(c:Company)
    RETURN c.name, COUNT(p) AS employees, AVG(p.salary) AS avg_salary
    ORDER BY employees DESC
""")

Supported functions: COUNT, SUM, AVG, MIN, MAX, COLLECT, COUNT(DISTINCT ...)


WITH Clauses

Chain multiple query stages with WITH:

Python
sql = graph.transpile("""
    MATCH (p:Person)-[:WORKS_AT]->(c:Company)
    WITH c, COUNT(p) AS employee_count
    WHERE employee_count > 100
    MATCH (c)-[:LOCATED_IN]->(city:City)
    RETURN c.name, employee_count, city.name
""")

UNION Queries

Combine multiple result sets:

Python
sql = graph.transpile("""
    MATCH (p:Person)-[:KNOWS]->(friend:Person)
    RETURN p.name AS source, friend.name AS target, 'KNOWS' AS type
    UNION
    MATCH (p:Person)-[:WORKS_AT]->(c:Company)
    RETURN p.name AS source, c.name AS target, 'WORKS_AT' AS type
""")

Execution on Databricks

If you provide a SparkSession, you can execute queries directly:

Python
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

graph = GraphContext(
    spark=spark,  # Enable execution
    nodes_table="catalog.schema.nodes",
    edges_table="catalog.schema.edges",
)
graph.set_types(node_types=["Person"], edge_types=["KNOWS"])

# Transpile and execute
sql = graph.transpile("MATCH (p:Person) RETURN p.name LIMIT 10")
df = spark.sql(sql)
df.show()

Low-Level API (Without GraphContext)

For complex scenarios where graph data is spread across multiple tables (not a simple Triple Store), or when you need full control, use the components directly:

Python
from gsql2rsql import OpenCypherParser, LogicalPlan, SQLRenderer
from gsql2rsql.common.schema import NodeSchema, EdgeSchema, EntityProperty
from gsql2rsql.renderer.schema_provider import SimpleSQLSchemaProvider, SQLTableDescriptor

# 1. Define schema (SimpleSQLSchemaProvider)
schema = SimpleSQLSchemaProvider()

person = NodeSchema(
    name="Person",
    properties=[
        EntityProperty(property_name="id", data_type=int),
        EntityProperty(property_name="name", data_type=str),
    ],
    node_id_property=EntityProperty(property_name="id", data_type=int)
)
schema.add_node(
    person,
    SQLTableDescriptor(
        table_name="catalog.schema.people",  # Separate table for Person nodes
        node_id_columns=["id"],
    )
)

knows = EdgeSchema(
    name="KNOWS",
    source_node_id="Person",
    sink_node_id="Person",
    source_id_property=EntityProperty(property_name="person_id", data_type=int),
    sink_id_property=EntityProperty(property_name="friend_id", data_type=int),
)
schema.add_edge(
    knows,
    SQLTableDescriptor(
        entity_id="Person@KNOWS@Person",
        table_name="catalog.schema.friendships",  # Separate table for KNOWS edges
    )
)

# 2. Transpile
parser = OpenCypherParser()
ast = parser.parse("MATCH (p:Person)-[:KNOWS]->(f:Person) RETURN p.name, f.name")
plan = LogicalPlan.process_query_tree(ast, schema)
plan.resolve(original_query="...")

renderer = SQLRenderer(db_schema_provider=schema)
sql = renderer.render_plan(plan)

This approach is useful when:

  • Different tables for different node/edge types (e.g., people, companies, friendships)
  • Custom column mappings beyond what GraphContext supports
  • Integration with existing schemas that don't follow Triple Store pattern

CLI Usage

gsql2rsql includes a command-line interface:

Bash
# Transpile a query
echo "MATCH (p:Person) RETURN p.name" | gsql2rsql translate --schema schema.json

# Interactive TUI
gsql2rsql tui --schema schema.json

OpenCypher Grammar Notes

gsql2rsql follows the standard OpenCypher grammar. Some patterns that might seem intuitive are actually invalid Cypher syntax.

WHERE Clause Placement

WHERE can only appear directly after MATCH or WITH clauses - not after UNWIND:

Python
# ❌ INVALID - WHERE cannot follow UNWIND directly
sql = graph.transpile("""
    MATCH (s)-[e:KNOWS*1..3]->(o)
    UNWIND e AS r
    WHERE r.weight > 5
    RETURN r.src, r.dst
""")
# Error: no viable alternative at input 'WHERE'

# ✅ CORRECT - Use WITH to enable WHERE filtering
sql = graph.transpile("""
    MATCH (s)-[e:KNOWS*1..3]->(o)
    UNWIND e AS r
    WITH r WHERE r.weight > 5
    RETURN r.src, r.dst
""")

This is standard Cypher behavior (same as Neo4j).

Reserved Words

Some words are reserved and cannot be used as variable names:

Reserved Word Used In Workaround
end CASE ... END Use target, endpoint, dest
start Legacy syntax Use source, origin, from_node
match MATCH clause Use matched, result
return RETURN clause Use result, output
Python
# ❌ INVALID - 'end' is a reserved word
sql = graph.transpile("""
    MATCH (start)-[:KNOWS]->(end)
    RETURN end.name
""")
# Error: no viable alternative at input '...)->(end'

# ✅ CORRECT - Use a different variable name
sql = graph.transpile("""
    MATCH (start)-[:KNOWS]->(target)
    RETURN target.name
""")

String Concatenation

Cypher uses + for string concatenation, but Spark SQL uses CONCAT(). Use the SQL function directly:

Python
# ❌ May fail in Spark - '+' not supported for strings
sql = graph.transpile("""
    MATCH (p:Person)
    RETURN p.first_name + ' ' + p.last_name AS full_name
""")

# ✅ CORRECT - Use CONCAT function
sql = graph.transpile("""
    MATCH (p:Person)
    RETURN CONCAT(p.first_name, ' ', p.last_name) AS full_name
""")

Limitations

  • Databricks new Runtime required for WITH RECURSIVE and HoF
  • Write operations not supported (CREATE, DELETE, SET)
  • Multi-label node syntax not yet supported:
    • (a:Person|Company) - pipe OR syntax causes parser error
    • (a:Person:Company) - colon AND syntax silently ignores additional labels
    • Workaround: Use WHERE a.node_type IN ['Person', 'Company']
  • Some Cypher features not yet implemented

Next Steps