User Guide¶
This guide covers everything you need to start using gsql2rsql to transpile OpenCypher queries to Databricks SQL.
Looking for examples?
After learning the basics, check out complete query examples:
- Fraud Detection - Fraud rings, money laundering patterns
- Credit Analysis - Risk assessment, guarantor chains
- All Features - 69 queries covering every feature
Installation¶
Requirements:
- Python 3.12+
- Databricks Runtime 15+ only needed to execute the generated SQL
Try It Now¶
No database needed! Just define a schema and generate SQL:
from gsql2rsql import GraphContext
# Table names are just strings - no database connection required
graph = GraphContext(
nodes_table="my_nodes",
edges_table="my_edges",
)
graph.set_types(
node_types=["Person", "Company"],
edge_types=["WORKS_AT"],
)
# Generate SQL from OpenCypher
sql = graph.transpile("""
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
RETURN p.node_id, c.node_id
""")
print(sql) # Copy this SQL to run on Databricks
Generated SQL Output
SELECT
_gsql2rsql_p_node_id AS node_id
,_gsql2rsql_c_node_id AS node_id
FROM (
SELECT
_left_0._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
,_left_0._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
,_left_0._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
,_right_0._gsql2rsql_c_node_id AS _gsql2rsql_c_node_id
FROM (
SELECT
_left_1._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
,_right_1._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
,_right_1._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
FROM (
SELECT
node_id AS _gsql2rsql_p_node_id
FROM
my_nodes
WHERE (node_type = 'Person')
) AS _left_1
INNER JOIN (
SELECT
src AS _gsql2rsql__anon1_src
,dst AS _gsql2rsql__anon1_dst
FROM
my_edges
WHERE (relationship_type = 'WORKS_AT')
) AS _right_1 ON
_left_1._gsql2rsql_p_node_id = _right_1._gsql2rsql__anon1_src
) AS _left_0
INNER JOIN (
SELECT
node_id AS _gsql2rsql_c_node_id
FROM
my_nodes
WHERE (node_type = 'Company')
) AS _right_0 ON
_right_0._gsql2rsql_c_node_id = _left_0._gsql2rsql__anon1_dst
) AS _proj
GraphContext: Full Configuration¶
GraphContext is the recommended API for graph data stored in the Triple Store pattern (one nodes table + one edges table). It eliminates ~100 lines of schema boilerplate.
With Node/Edge Attributes¶
from gsql2rsql import GraphContext
# Define schema with additional attributes
graph = GraphContext(
nodes_table="catalog.schema.nodes",
edges_table="catalog.schema.edges",
extra_node_attrs={"name": str, "age": int, "score": float},
extra_edge_attrs={"weight": float, "timestamp": str},
)
graph.set_types(
node_types=["Person", "Company", "Account"],
edge_types=["KNOWS", "WORKS_AT", "OWNS"],
)
# Now you can use those attributes in queries
sql = graph.transpile("""
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
WHERE c.industry = 'Technology'
RETURN p.name, c.name AS company
LIMIT 100
""")
print(sql)
Generated SQL
SELECT
_gsql2rsql_p_name AS name
,_gsql2rsql_c_name AS company
FROM (
SELECT
_left_0._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
,_left_0._gsql2rsql_p_name AS _gsql2rsql_p_name
,_left_0._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
,_left_0._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
,_right_0._gsql2rsql_c_node_id AS _gsql2rsql_c_node_id
,_right_0._gsql2rsql_c_name AS _gsql2rsql_c_name
,_right_0._gsql2rsql_c_industry AS _gsql2rsql_c_industry
FROM (
SELECT
_left_1._gsql2rsql_p_node_id AS _gsql2rsql_p_node_id
,_left_1._gsql2rsql_p_name AS _gsql2rsql_p_name
,_right_1._gsql2rsql__anon1_src AS _gsql2rsql__anon1_src
,_right_1._gsql2rsql__anon1_dst AS _gsql2rsql__anon1_dst
FROM (
SELECT
node_id AS _gsql2rsql_p_node_id
,name AS _gsql2rsql_p_name
FROM
catalog.schema.nodes
WHERE (node_type = 'Person')
) AS _left_1
INNER JOIN (
SELECT
src AS _gsql2rsql__anon1_src
,dst AS _gsql2rsql__anon1_dst
FROM
catalog.schema.edges
WHERE (relationship_type = 'WORKS_AT')
) AS _right_1 ON
_left_1._gsql2rsql_p_node_id = _right_1._gsql2rsql__anon1_src
) AS _left_0
INNER JOIN (
SELECT
node_id AS _gsql2rsql_c_node_id
,name AS _gsql2rsql_c_name
,industry AS _gsql2rsql_c_industry
FROM
catalog.schema.nodes
WHERE (node_type = 'Company') AND ((industry) = ('Technology'))
) AS _right_0 ON
_right_0._gsql2rsql_c_node_id = _left_0._gsql2rsql__anon1_dst
) AS _proj
LIMIT 100
GraphContext Parameters¶
| Parameter | Type | Description |
|---|---|---|
nodes_table | str | Full path to nodes table (e.g., catalog.schema.nodes) |
edges_table | str | Full path to edges table |
node_type_col | str | Column for node type (default: "node_type") |
edge_type_col | str | Column for edge type (default: "relationship_type") |
node_id_col | str | Column for node ID (default: "node_id") |
edge_src_col | str | Column for edge source (default: "src") |
edge_dst_col | str | Column for edge destination (default: "dst") |
extra_node_attrs | dict | Additional node properties {name: type} |
extra_edge_attrs | dict | Additional edge properties {name: type} |
spark | SparkSession | Optional: for auto-discovery and execution |
optimize_dead_tables | bool | Enable dead table elimination optimization (default: True) |
Automatic System Properties¶
GraphContext automatically exposes system properties for use in queries, without needing to declare them in extra_node_attrs or extra_edge_attrs:
Node system properties:
| Property | Default Column | Description |
|---|---|---|
node_id | node_id | Unique node identifier |
node_type | node_type | Node label/type |
Edge system properties:
| Property | Default Column | Description |
|---|---|---|
src | src | Source node ID |
dst | dst | Target node ID |
relationship_type | relationship_type | Edge label/type |
This allows queries like:
# Filter by node_type without declaring it in extra_node_attrs
sql = graph.transpile("""
MATCH (a)-[r]->(b)
WHERE a.node_type IN ['Person', 'Company']
RETURN a.node_id, r.relationship_type, b.node_id
""")
Expected Table Structure¶
Nodes table:
| Column | Description |
|---|---|
node_id | Unique node identifier |
node_type | Node label (Person, Company, etc.) |
name, age, ... | Node properties |
Edges table:
| Column | Description |
|---|---|
src | Source node ID |
dst | Destination node ID |
relationship_type | Edge label (KNOWS, WORKS_AT, etc.) |
weight, ... | Edge properties |
No-Label Nodes (Wildcard Matching)¶
GraphContext automatically enables no-label support, allowing nodes without explicit labels in queries:
# Node 'a' has no label - matches ANY node type
sql = graph.transpile("""
MATCH (a)-[:WORKS_AT]->(c:Company)
RETURN a, c.name
""")
This is useful when:
- You don't know or care about the source node type
- You want to match multiple node types at once
- You're exploring relationships without type constraints
Performance Impact
No-label nodes cause full table scans on the nodes table (no WHERE node_type = '...' filter). Use explicit labels whenever possible for production queries.
Variable-Length Paths¶
One of gsql2rsql's most powerful features is support for variable-length paths using WITH RECURSIVE.
sql = graph.transpile("""
MATCH path = (origin:Person {id: 12345})-[:KNOWS*1..3]->(friend:Person)
RETURN friend.name, length(path) AS hops
ORDER BY hops
""")
Key features:
- Depth bounds:
*1..3means 1 to 3 hops - Cycle detection: Automatic
ARRAY_CONTAINSchecks prevent infinite loops - Path functions:
length(path),nodes(path),relationships(path) - Path variable: Captures the entire path for analysis
Bidirectional BFS Optimization¶
When BOTH source AND target have equality filters on ID, bidirectional BFS can enable queries that would otherwise fail due to Spark's recursion limits.
sql = graph.transpile(
"""
MATCH path = (a:Person)-[:KNOWS*1..4]->(b:Person)
WHERE a.node_id = 'alice' AND b.node_id = 'dave'
RETURN nodes(path) AS path_nodes
""",
bidirectional_mode="recursive" # Enable optimization
)
Generated SQL (bidirectional off vs recursive)
WITH RECURSIVE
paths_1 AS (
-- Base case: direct edges (depth = 1)
SELECT
e.src AS start_node,
e.dst AS end_node,
1 AS depth,
ARRAY(e.src, e.dst) AS path,
ARRAY(NAMED_STRUCT('src', e.src, 'dst', e.dst, 'relationship_type', e.relationship_type)) AS path_edges,
ARRAY(e.src) AS visited
FROM graph.edges e
JOIN graph.nodes src ON src.node_id = e.src
WHERE (relationship_type = 'KNOWS') AND (src.node_id) = ('alice')
UNION ALL
-- Recursive case: extend paths
SELECT
p.start_node,
e.dst AS end_node,
p.depth + 1 AS depth,
CONCAT(p.path, ARRAY(e.dst)) AS path,
ARRAY_APPEND(p.path_edges, NAMED_STRUCT('src', e.src, 'dst', e.dst, 'relationship_type', e.relationship_type)) AS path_edges,
CONCAT(p.visited, ARRAY(e.src)) AS visited
FROM paths_1 p
JOIN graph.edges e
ON p.end_node = e.src
WHERE p.depth < 4
AND NOT ARRAY_CONTAINS(p.visited, e.dst)
AND (relationship_type = 'KNOWS')
)
SELECT
_gsql2rsql_path_id AS path_nodes
FROM (
SELECT
sink.node_id AS _gsql2rsql_b_node_id
,sink.node_type AS _gsql2rsql_b_node_type
,source.node_id AS _gsql2rsql_a_node_id
,source.node_type AS _gsql2rsql_a_node_type
,p.start_node
,p.end_node
,p.depth
,p.path AS _gsql2rsql_path_id
,p.path_edges AS _gsql2rsql_path_edges
FROM paths_1 p
JOIN graph.nodes sink
ON sink.node_id = p.end_node
JOIN graph.nodes source
ON source.node_id = p.start_node
WHERE p.depth >= 1 AND p.depth <= 4 AND sink.node_type = 'Person' AND source.node_type = 'Person' AND (sink.node_id) = ('dave')
) AS _proj
WITH RECURSIVE
forward_paths_1 AS (
-- Depth 0: src node itself (for meeting with backward)
SELECT
src.node_id AS current_node,
0 AS depth,
ARRAY(src.node_id) AS path,
CAST(ARRAY() AS ARRAY<STRUCT<src: STRING, dst: STRING>>) AS path_edges
FROM graph.nodes src
WHERE (src.node_id) = ('alice')
UNION ALL
-- Depth 1+: explore edges from src
SELECT
e.dst AS current_node,
1 AS depth,
ARRAY(e.src, e.dst) AS path,
ARRAY(STRUCT(e.src AS src, e.dst AS dst)) AS path_edges
FROM graph.edges e
JOIN graph.nodes src ON src.node_id = e.src
WHERE (relationship_type = 'KNOWS') AND (src.node_id) = ('alice')
UNION ALL
-- Recursive case: extend forward
SELECT
e.dst AS current_node,
f.depth + 1 AS depth,
CONCAT(f.path, ARRAY(e.dst)) AS path,
CONCAT(f.path_edges, ARRAY(STRUCT(e.src AS src, e.dst AS dst))) AS path_edges
FROM forward_paths_1 f
JOIN graph.edges e
ON f.current_node = e.src
WHERE f.depth < 2
AND NOT ARRAY_CONTAINS(f.path, e.dst)
AND (relationship_type = 'KNOWS')
),
backward_paths_1 AS (
-- Depth 0: tgt node itself (for meeting with forward)
SELECT
tgt.node_id AS current_node,
0 AS depth,
ARRAY(tgt.node_id) AS path,
CAST(ARRAY() AS ARRAY<STRUCT<src: STRING, dst: STRING>>) AS path_edges
FROM graph.nodes tgt
WHERE (tgt.node_id) = ('dave')
UNION ALL
-- Depth 1+: explore edges from tgt
SELECT
e.src AS current_node,
1 AS depth,
ARRAY(e.src, e.dst) AS path,
ARRAY(STRUCT(e.src AS src, e.dst AS dst)) AS path_edges
FROM graph.edges e
JOIN graph.nodes tgt ON tgt.node_id = e.dst
WHERE (relationship_type = 'KNOWS') AND (tgt.node_id) = ('dave')
UNION ALL
-- Recursive case: extend backward
SELECT
e.src AS current_node,
b.depth + 1 AS depth,
CONCAT(ARRAY(e.src), b.path) AS path,
CONCAT(ARRAY(STRUCT(e.src AS src, e.dst AS dst)), b.path_edges) AS path_edges
FROM backward_paths_1 b
JOIN graph.edges e
ON b.current_node = e.dst
WHERE b.depth < 2
AND NOT ARRAY_CONTAINS(b.path, e.src)
AND (relationship_type = 'KNOWS')
),
paths_1 AS (
-- Intersection: paths that meet in the middle
-- Use DISTINCT to deduplicate paths found via different meeting points
SELECT DISTINCT
f.path[0] AS start_node,
b.path[SIZE(b.path) - 1] AS end_node,
f.depth + b.depth AS depth,
CONCAT(SLICE(f.path, 1, SIZE(f.path) - 1), b.path) AS path,
CONCAT(f.path_edges, b.path_edges) AS path_edges
FROM forward_paths_1 f
JOIN backward_paths_1 b
ON f.current_node = b.current_node
WHERE f.depth + b.depth >= 1
AND f.depth + b.depth <= 4
AND SIZE(ARRAY_INTERSECT(SLICE(f.path, 1, SIZE(f.path) - 1), b.path)) = 0
)
SELECT
_gsql2rsql_path_id AS path_nodes
FROM (
SELECT
sink.node_id AS _gsql2rsql_b_node_id
,sink.node_type AS _gsql2rsql_b_node_type
,source.node_id AS _gsql2rsql_a_node_id
,source.node_type AS _gsql2rsql_a_node_type
,p.start_node
,p.end_node
,p.depth
,p.path AS _gsql2rsql_path_id
,p.path_edges AS _gsql2rsql_path_edges
FROM paths_1 p
JOIN graph.nodes sink
ON sink.node_id = p.end_node
JOIN graph.nodes source
ON source.node_id = p.start_node
WHERE p.depth >= 1 AND p.depth <= 4 AND sink.node_type = 'Person' AND source.node_type = 'Person' AND (sink.node_id) = ('dave')
) AS _proj
Why use it:
- Unidirectional BFS explores ALL paths from source (exponential growth)
- Bidirectional BFS explores from both ends and prunes paths that don't meet
- For small graphs: bidirectional has ~20-30% overhead
- For large graphs: unidirectional fails (hits
maxRowsPerIterationlimit), bidirectional succeeds
Modes:
| Mode | Description |
|---|---|
"off" | Standard unidirectional BFS |
"recursive" | WITH RECURSIVE forward/backward CTEs (default) |
"unrolling" | Unrolled CTEs (best for depth ≤6) |
"auto" | Auto-select based on max_hops |
Inline Filters¶
Use inline property filters for better performance:
# Inline filter (optimized - pushed to source scan)
sql = graph.transpile("""
MATCH (p:Person {status: 'active'})-[:KNOWS]->(friend)
RETURN p.name, friend.name
""")
# Equivalent WHERE clause (less optimized)
sql = graph.transpile("""
MATCH (p:Person)-[:KNOWS]->(friend)
WHERE p.status = 'active'
RETURN p.name, friend.name
""")
Inline filters are pushed down to the data source scan, reducing the amount of data processed.
Undirected Relationships¶
Use -[:REL]- (without arrow) for undirected relationships:
sql = graph.transpile("""
MATCH (a:Person)-[:KNOWS]-(b:Person)
WHERE a.id = 123
RETURN b.name
""")
This generates a UNION ALL query to match both directions efficiently.
Aggregations¶
Standard aggregation functions are supported:
sql = graph.transpile("""
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
RETURN c.name, COUNT(p) AS employees, AVG(p.salary) AS avg_salary
ORDER BY employees DESC
""")
Supported functions: COUNT, SUM, AVG, MIN, MAX, COLLECT, COUNT(DISTINCT ...)
WITH Clauses¶
Chain multiple query stages with WITH:
sql = graph.transpile("""
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
WITH c, COUNT(p) AS employee_count
WHERE employee_count > 100
MATCH (c)-[:LOCATED_IN]->(city:City)
RETURN c.name, employee_count, city.name
""")
UNION Queries¶
Combine multiple result sets:
sql = graph.transpile("""
MATCH (p:Person)-[:KNOWS]->(friend:Person)
RETURN p.name AS source, friend.name AS target, 'KNOWS' AS type
UNION
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
RETURN p.name AS source, c.name AS target, 'WORKS_AT' AS type
""")
Execution on Databricks¶
If you provide a SparkSession, you can execute queries directly:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
graph = GraphContext(
spark=spark, # Enable execution
nodes_table="catalog.schema.nodes",
edges_table="catalog.schema.edges",
)
graph.set_types(node_types=["Person"], edge_types=["KNOWS"])
# Transpile and execute
sql = graph.transpile("MATCH (p:Person) RETURN p.name LIMIT 10")
df = spark.sql(sql)
df.show()
Low-Level API (Without GraphContext)¶
For complex scenarios where graph data is spread across multiple tables (not a simple Triple Store), or when you need full control, use the components directly:
from gsql2rsql import OpenCypherParser, LogicalPlan, SQLRenderer
from gsql2rsql.common.schema import NodeSchema, EdgeSchema, EntityProperty
from gsql2rsql.renderer.schema_provider import SimpleSQLSchemaProvider, SQLTableDescriptor
# 1. Define schema (SimpleSQLSchemaProvider)
schema = SimpleSQLSchemaProvider()
person = NodeSchema(
name="Person",
properties=[
EntityProperty(property_name="id", data_type=int),
EntityProperty(property_name="name", data_type=str),
],
node_id_property=EntityProperty(property_name="id", data_type=int)
)
schema.add_node(
person,
SQLTableDescriptor(
table_name="catalog.schema.people", # Separate table for Person nodes
node_id_columns=["id"],
)
)
knows = EdgeSchema(
name="KNOWS",
source_node_id="Person",
sink_node_id="Person",
source_id_property=EntityProperty(property_name="person_id", data_type=int),
sink_id_property=EntityProperty(property_name="friend_id", data_type=int),
)
schema.add_edge(
knows,
SQLTableDescriptor(
entity_id="Person@KNOWS@Person",
table_name="catalog.schema.friendships", # Separate table for KNOWS edges
)
)
# 2. Transpile
parser = OpenCypherParser()
ast = parser.parse("MATCH (p:Person)-[:KNOWS]->(f:Person) RETURN p.name, f.name")
plan = LogicalPlan.process_query_tree(ast, schema)
plan.resolve(original_query="...")
renderer = SQLRenderer(db_schema_provider=schema)
sql = renderer.render_plan(plan)
This approach is useful when:
- Different tables for different node/edge types (e.g.,
people,companies,friendships) - Custom column mappings beyond what GraphContext supports
- Integration with existing schemas that don't follow Triple Store pattern
CLI Usage¶
gsql2rsql includes a command-line interface:
# Transpile a query
echo "MATCH (p:Person) RETURN p.name" | gsql2rsql translate --schema schema.json
# Interactive TUI
gsql2rsql tui --schema schema.json
OpenCypher Grammar Notes¶
gsql2rsql follows the standard OpenCypher grammar. Some patterns that might seem intuitive are actually invalid Cypher syntax.
WHERE Clause Placement¶
WHERE can only appear directly after MATCH or WITH clauses - not after UNWIND:
# ❌ INVALID - WHERE cannot follow UNWIND directly
sql = graph.transpile("""
MATCH (s)-[e:KNOWS*1..3]->(o)
UNWIND e AS r
WHERE r.weight > 5
RETURN r.src, r.dst
""")
# Error: no viable alternative at input 'WHERE'
# ✅ CORRECT - Use WITH to enable WHERE filtering
sql = graph.transpile("""
MATCH (s)-[e:KNOWS*1..3]->(o)
UNWIND e AS r
WITH r WHERE r.weight > 5
RETURN r.src, r.dst
""")
This is standard Cypher behavior (same as Neo4j).
Reserved Words¶
Some words are reserved and cannot be used as variable names:
| Reserved Word | Used In | Workaround |
|---|---|---|
end | CASE ... END | Use target, endpoint, dest |
start | Legacy syntax | Use source, origin, from_node |
match | MATCH clause | Use matched, result |
return | RETURN clause | Use result, output |
# ❌ INVALID - 'end' is a reserved word
sql = graph.transpile("""
MATCH (start)-[:KNOWS]->(end)
RETURN end.name
""")
# Error: no viable alternative at input '...)->(end'
# ✅ CORRECT - Use a different variable name
sql = graph.transpile("""
MATCH (start)-[:KNOWS]->(target)
RETURN target.name
""")
String Concatenation¶
Cypher uses + for string concatenation, but Spark SQL uses CONCAT(). Use the SQL function directly:
# ❌ May fail in Spark - '+' not supported for strings
sql = graph.transpile("""
MATCH (p:Person)
RETURN p.first_name + ' ' + p.last_name AS full_name
""")
# ✅ CORRECT - Use CONCAT function
sql = graph.transpile("""
MATCH (p:Person)
RETURN CONCAT(p.first_name, ' ', p.last_name) AS full_name
""")
Limitations¶
- Databricks new Runtime required for
WITH RECURSIVEand HoF - Write operations not supported (
CREATE,DELETE,SET) - Multi-label node syntax not yet supported:
(a:Person|Company)- pipe OR syntax causes parser error(a:Person:Company)- colon AND syntax silently ignores additional labels- Workaround: Use
WHERE a.node_type IN ['Person', 'Company']
- Some Cypher features not yet implemented
Next Steps¶
- Examples Gallery - query examples