Contributing Guide¶

This guide covers architecture, development workflow, and contribution guidelines.

Why Contribute?¶

Rich Error Messages¶

One of the key features of this transpiler is its developer-friendly error messages. When something goes wrong, you get detailed context to quickly identify and fix the issue:

╔══════════════════════════════════════════════════════════════════════════════╗
║ ColumnResolutionError: Variable 'rels' is not defined                        ║
╚══════════════════════════════════════════════════════════════════════════════╝

━━━ Query ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

    1 │ MATCH path = (root:Vertex)-[rels:REL*1..5]-(n:Vertex) WHERE root.node_id = '1234' RETURN rels
      │                             ▲
      │                             └── ERROR: Variable 'rels' is not defined
    2 │

━━━ Available Variables (Scope Level 0) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Name         Type      Data Type   Defined At              Properties
  ───────────────────────────────────────────────────────────────────────────
  root         entity   Vertex       MATCH (root:Vertex)      node_type, metadata, node_id
  path         path     PATH         MATCH path = ...         -
  n            entity   Vertex       MATCH (n:Vertex)         node_type, metadata, node_id

━━━ Suggestions ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Did you mean 'root'? (3 characters difference)

━━━ Hints ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  💡 Make sure 'rels' is defined in a MATCH clause before use.
     Variables must be defined before they can be referenced in WHERE, WITH, or RETURN clauses.

━━━ Debug Information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Operator:         ProjectionOperator (id=6)
  Resolution Phase: expression_resolution
  Symbol Table:
    Symbol Table Dump:
      Scope 0 (global):
        root: entity(Vertex) @ scope 0
        path: path(PATH) @ scope 0
        n: entity(Vertex) @ scope 0

Error messages include: - Visual pointer to the exact error location in the query - Available variables at that scope with their types and properties - Suggestions for typos (using Levenshtein distance) - Hints explaining why the error occurred - Debug info for advanced troubleshooting

Architecture Overview¶

High-Level Dataflow¶

graph TD
    A[OpenCypher Query String] --> B[Phase 1: Parser]
    B --> C[Abstract Syntax Tree]
    C --> D[Phase 2: Planner]
    D --> E[Logical Operator Tree + SymbolTable]
    E --> F[Phase 3: Optimizer]
    F --> G[Optimized Logical Plan]
    G --> H[Phase 4: Resolver]
    H --> I[ResolutionResult]
    I --> J[Phase 5: Renderer]
    J --> K[Databricks Databricks SQL]

    style B fill:#e1f5ff
    style D fill:#fff4e1
    style F fill:#ffe1f5
    style H fill:#e1ffe1
    style J fill:#ffe1e1

Design Principle¶

Each phase has a single responsibility and does not perform the responsibilities of other phases:

1. Parser: Syntax only (what is valid Cypher?)
2. Planner: Semantics only (what does it mean logically?)
3. Resolver: Validation only (do all references exist?)
4. Renderer: Implementation only (how to generate SQL?)

This prevents: - Parser from needing schema knowledge - Planner from validating column references - Renderer from making semantic decisions

Phase Details¶

Phase 1: Parser (Lexical/Syntactic Analysis)¶

Location: src/gsql2rsql/parser/

Responsibility: Convert OpenCypher query string to Abstract Syntax Tree (AST). Validates syntax only — does NOT validate semantics, resolve references, or access schema.

Does NOT:

• ❌ Validate semantics
• ❌ Resolve references
• ❌ Access schema
• ❌ Perform type checking
• ❌ Validate property names

Rules:

• Parser MUST NOT import from planner/, renderer/, or common/schema.py
• Parser MUST NOT call graph schema provider
• Parser MUST only validate syntax (grammar rules)

Phase 2: Planner (Logical Operator Construction)¶

Location: src/gsql2rsql/planner/

Responsibility: Convert AST to logical relational algebra. Builds symbol table tracking variable definitions and scopes. Does NOT resolve column references or validate property access.

Does NOT:

• ❌ Resolve column references
• ❌ Validate property access
• ❌ Generate SQL
• ❌ Query database schema

Rules:

• Planner CAN import from parser/ (uses AST)
• Planner CAN import from common/schema.py (uses GraphSchema)
• Planner MUST NOT import from renderer/
• Planner MUST NOT perform column resolution (that's Phase 4)

Phase 3: Optimizer (Conservative Transformations)¶

Location: src/gsql2rsql/planner/subquery_optimizer.py

Responsibility: Apply conservative transformations to reduce SQL nesting. Only flattens patterns guaranteed to preserve semantics.

Safe to Flatten: - ✅ Selection → Projection: WHERE can be pushed before SELECT - ✅ Selection → Selection: Multiple WHERE clauses can be merged

NEVER Flattened: - ❌ Projection → Projection: Aliases must stay in separate subqueries - ❌ Anything involving aggregation boundaries: Aggregation semantics are fragile

Configuration: Enabled by default. Disable with --no-optimize CLI flag.

Phase 4: Resolver (Column Reference Validation)¶

Location: src/gsql2rsql/planner/column_resolver.py

Responsibility: Validate ALL column references before rendering. Query schema for entity properties. Build resolution context for SQL generation.

Does NOT:

• ❌ Generate SQL
• ❌ Modify logical plan structure
• ❌ Perform optimizations

Rules:

• Resolver CAN import from parser/, planner/, common/
• Resolver MUST NOT import from renderer/
• Resolver MUST validate ALL column refs before SQL generation
• Resolver MUST provide rich error messages with suggestions

Phase 5: Renderer (SQL Generation)¶

Location: src/gsql2rsql/renderer/sql_renderer.py

Responsibility: Generate Databricks Databricks SQL from logical plan using pre-resolved column references. Handle SQL dialect specifics.

Does NOT:

• ❌ Resolve columns
• ❌ Validate references
• ❌ Make semantic decisions

Rules:

• Renderer CAN import from all phases (uses everything)
• Renderer MUST use ResolutionResult for all column refs
• Renderer MUST NOT resolve columns itself
• Renderer MUST NOT perform semantic validation

Quick Start for Contributors¶

1. Set Up Development Environment¶

Option A: Using Makefile (Recommended)¶

# Clone repository
git clone https://github.com/your-org/cyper2dsql.git
cd cyper2dsql/python

# Create virtual environment and install dev dependencies
make venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
make install-dev

Option B: Using uv directly (if make is not available)¶

# Clone repository
git clone https://github.com/your-org/cyper2dsql.git
cd cyper2dsql/python

# Create virtual environment with uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies (dev mode)
uv sync --extra dev
uv pip install -e ".[dev]"

2. Verify Installation¶

Using Makefile¶

make test-no-pyspark  # Run fast test suite (excludes PySpark)
make check            # Run linter + type checker
make cli-example      # Transpile example query

Using uv directly¶

uv run pytest tests/ -v --ignore=tests/test_examples_with_pyspark.py --ignore=tests/test_pyspark_basic.py
uv run ruff check src/ tests/
uv run pyright src/
echo "MATCH (p:Person)-[:KNOWS]->(f:Person) RETURN p.name, f.name" | uv run gsql2rsql transpile -s examples/schema.json

Available Make Commands¶

Run make help to see all available commands:

3. Make Changes¶

Before coding: 1. Read phase details (critical!) 2. Identify which phase your change affects (parser, planner, resolver, or renderer) 3. Add tests first (TDD recommended)

Development loop:

# 1. Make changes to src/
# 2. Run fast tests
make test-no-pyspark

# 3. Run all checks
make check

# 4. Update golden files if SQL output changes
make dump-sql-save ID=01 NAME=simple_node_lookup

# 5. Run full test suite
make test

# 1. Make changes to src/
# 2. Run fast tests
uv run pytest tests/ -v --ignore=tests/test_examples_with_pyspark.py --ignore=tests/test_pyspark_basic.py

# 3. Run lint and type checks
uv run ruff check src/ tests/
uv run pyright src/

# 4. Run full test suite
uv run pytest tests/ -n 6

4. Submit Changes¶

# Format code
make format

# Run all checks
make check

# Run full test suite (including PySpark)
make test

# Commit with descriptive message
git add .
git commit -m "feat: add support for map projection in RETURN clause"

# Push and create PR
git push origin feature/map-projection

# Format code
uv run ruff format src/ tests/

# Run lint and type checks
uv run ruff check src/ tests/
uv run pyright src/

# Run full test suite
uv run pytest tests/ -n 6

# Commit with descriptive message
git add .
git commit -m "feat: add support for map projection in RETURN clause"

# Push and create PR
git push origin feature/map-projection

Architectural Examples¶

Valid vs Invalid Changes¶

Parser Example¶

Valid Change - Adding a new AST node type:

# ✅ Adding a new AST node type for a new Cypher construct
class QueryExpressionPatternComprehension(QueryExpression):
    pattern: QueryPattern
    where_clause: Optional[WhereClause]
    projection: QueryExpression

Invalid Change - Parser accessing schema:

# ❌ WRONG: Parser accessing schema
class CypherVisitor:
    def visitPropertyExpression(self, ctx):
        entity_name = self._get_entity_name(ctx)
        # ❌ WRONG: Don't validate property existence here
        if not self.schema.has_property(entity_name, property_name):
            raise Exception("Property not found")

Planner Example¶

Valid Change - Adding a new logical operator:

# ✅ Adding a new logical operator
class WindowOperator(LogicalOperator):
    """Represents a window function (OVER clause)."""
    partition_by: list[str]
    order_by: list[OrderByItem]
    window_function: WindowFunction

Invalid Change - Planner resolving column references:

# ❌ WRONG: Planner resolving column references
class LogicalPlan:
    def _process_projection(self, projection: ProjectionItem):
        # ❌ WRONG: Don't resolve column refs during planning
        resolved_ref = self._resolve_column_reference(projection.expression)
        # Column resolution belongs in Phase 4 (Resolver)

Renderer Example¶

Valid Change - Adding support for new SQL function:

# ✅ Adding support for new SQL function
class SQLRenderer:
    def _render_function(self, func: Function, args: list[str]) -> str:
        if func == Function.RTRIM:
            # New function mapping
            return f"RTRIM({', '.join(args)})"
        # ... existing functions

Invalid Change - Renderer resolving columns:

# ❌ WRONG: Renderer resolving columns
class SQLRenderer:
    def _render_property_access(self, entity: str, property: str) -> str:
        # ❌ WRONG: Don't resolve property here
        if not self.schema.has_property(entity, property):
            raise Exception("Property not found")
        # Resolution should already be done in Phase 4

Common Development Tasks¶

Adding a New Cypher Feature¶

Example: Add support for range() function

1. Phase 1 (Parser): Add to grammar or AST if needed

   Python

   # If new function, add to Function enum in operators.py
   class Function(Enum):
       RANGE = "range"  # Generate sequence of integers
   

2. Phase 2 (Planner): Handle in operator construction (if needed)

   Python

   # Usually functions are just expressions, no special operator needed
   

3. Phase 4 (Resolver): Type checking (if needed)

   Python

   # Add type evaluation rule
   def _evaluate_function_type(self, func: Function, args: list[DataType]) -> DataType:
       if func == Function.RANGE:
           return DataType.LIST_INT
   

4. Phase 5 (Renderer): Add SQL generation

   Python

   class SQLRenderer:
       def _render_function(self, func: Function, args: list[str]) -> str:
           if func == Function.RANGE:
               # Databricks: sequence(start, stop, step)
               return f"sequence({args[0]}, {args[1]})"
   

5. Add Tests: Golden file test + unit tests

   Bash

   # Create test_46_range_function.py
   # Generate golden file
   make dump-sql-save ID=46 NAME=range_function
   

Fixing a Bug¶

Example: Fix incorrect null handling in OPTIONAL MATCH

1. Write failing test first (TDD)

   Python

   # tests/test_optional_match_null_bug.py
   def test_optional_match_null_handling():
       cypher = "MATCH (p:Person) OPTIONAL MATCH (p)-[:KNOWS]->(f) RETURN p.name, f.name"
       sql = transpile(cypher)
       # Should use COALESCE for f.name
       assert "COALESCE" in sql
   

2. Run test (should fail)

   Bash

   pytest tests/test_optional_match_null_bug.py -v
   

3. Identify the phase where the bug is (use --explain-scopes for debugging)

   Bash

   echo "MATCH (p) OPTIONAL MATCH (p)-[:KNOWS]->(f) RETURN p.name, f.name" | \
     uv run gsql2rsql transpile -s examples/schema.json --explain-scopes
   

4. Fix the bug in the appropriate phase

   Python

   # src/gsql2rsql/renderer/sql_renderer.py
   def _render_optional_property(self, ref: ResolvedColumnRef) -> str:
       if ref.is_from_optional_match:
           # ✅ Add COALESCE for null handling
           return f"COALESCE({ref.sql_column_name}, NULL)"
       return ref.sql_column_name
   

5. Run tests (should pass now)

   Bash

   pytest tests/test_optional_match_null_bug.py -v
   make test-no-pyspark
   

6. Update golden files if SQL output changed

   Bash

   make diff-all  # Review changes
   make dump-sql-save ID=09 NAME=optional_match  # Update if correct
   

Debugging Transpilation Issues¶

Step 1: Isolate the query

# Save problematic query to file
echo "MATCH (n:Node) WHERE n.prop > 10 RETURN n" > debug_query.cypher

Step 2: Inspect AST

uv run gsql2rsql parse -i debug_query.cypher

Step 3: Inspect logical plan

# In Python REPL or script
from gsql2rsql import OpenCypherParser, LogicalPlan
from gsql2rsql.renderer.schema_provider import SimpleSQLSchemaProvider

parser = OpenCypherParser()
ast = parser.parse(open("debug_query.cypher").read())

schema = SimpleSQLSchemaProvider()
# ... add schema

plan = LogicalPlan.process_query_tree(ast, schema)
print(plan.dump_graph())  # Visualize operator tree

Step 4: Check scopes

uv run gsql2rsql transpile -s examples/schema.json -i debug_query.cypher --explain-scopes

Testing Error Messages¶

The transpiler has rich error messages. To test error behavior:

# Test a query that should produce an error
make test-recursive-query

# Or manually test with a bad query
echo "MATCH (n:Person) RETURN undefined_var" | uv run gsql2rsql transpile -s examples/schema.json

Error messages are generated in src/gsql2rsql/common/exceptions.py. When adding new error types:

1. Create a descriptive error class
2. Include context (query, position, available variables)
3. Add helpful suggestions (typo detection, hints)
4. Test the error message visually

Inspiration and Design Differences¶

gsql2rsql was inspired by the Microsoft openCypherTranspiler, a C# project for transpiling OpenCypher to T-SQL (now discontinued). While the core idea is similar—translating Cypher graph queries to SQL—gsql2rsql introduces several key architectural differences:

• Stricter Phase Separation: gsql2rsql enforces a much stronger separation between the phases of the transpiler pipeline (Parser, Planner, Resolver, Renderer). Each phase has a single responsibility, and the renderer is intentionally kept as "dumb" as possible, only emitting SQL from fully-resolved logical plans. This separation makes the codebase easier to maintain, test, and extend.

• Human-Friendly Debugging: The architecture is designed for transparency and developer experience. For example, error messages during development are rich and actionable, showing available variables, suggestions, and hints. See the example below:

  Text Only

  Makefile:55: warning: ignoring old recipe for target 'test-pyspark-quick'
  Testing recursive query transpilation...
  ╔══════════════════════════════════════════════════════════════════════════════╗
  ║ ColumnResolutionError: Variable 'rels' is not defined                        ║
  ╚══════════════════════════════════════════════════════════════════════════════╝
  
  ━━━ Query ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
      1 │ MATCH path = (root:Vertex)-[rels:REL*1..5]-(n:Vertex) WHERE root.node_id = '1234_algo' AND n.node_type = 'node_type' AND NONE(r IN rels WHERE r.relationship_type IN ['a', 'b']) RETURN rels AS edges, n AS vertex_info
        │                             ▲
        │                             └── ERROR: Variable 'rels' is not defined
      2 │
  
  ━━━ Available Variables (Scope Level 0) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
    Name         Type      Data Type   Defined At              Properties
    ───────────────────────────────────────────────────────────────────────────
    root         entity   Vertex       MATCH (root:Vertex)      node_type, metadata, node_id
    path         path     PATH         MATCH path = ...         -
    n            entity   Vertex       MATCH (n:Vertex)         node_type, metadata, node_id
    edges        value    unknown      RETURN/WITH AS edges     -
    vertex_info  value    Vertex       RETURN/WITH AS vertex_info -
  
  ━━━ Suggestions ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
    • Did you mean 'root'? (3 characters difference)
  
  ━━━ Hints ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
    💡 Make sure 'rels' is defined in a MATCH clause before use.
       Variables must be defined before they can be referenced in WHERE, WITH, or RETURN clauses.
  
  ━━━ Debug Information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
    Operator:         ProjectionOperator (id=6)
    Resolution Phase: expression_resolution
    Symbol Table:
      Symbol Table Dump:
        Scope 0 (global):
          root: entity(Vertex) @ scope 0
          path: path(PATH) @ scope 0
          n: entity(Vertex) @ scope 0
          edges: value(unknown) @ scope 0
          vertex_info: value(Vertex) @ scope 0