SemBench

1 Run the Benchmark

The CLI entry point is src/run.py. You specify which system, scenario, and queries to run. The script dynamically loads the correct runner class for each system using get_runner_class().

2 The Runner Base Class

Every system runner inherits from GenericRunner (located at src/runner/generic_runner.py). It defines the abstract interface (get_system_name(), execute_query()), sets up paths to data, queries, and results, and provides load_data() for reading CSVs. It also provides two query-discovery mechanisms: _discover_query_impl(query_id) finds a Python method named _execute_q{id} via reflection (used by Code mode), and _discover_query_text(query_id) locates a Q{id}.sql file in the system’s query directory (used by SQL mode). You should not need to modify this file.

3 Build Your System’s Base Runner

Each system should implement its own intermediate base runner between GenericRunner and the per-scenario runners. Create it at src/runner/generic_{system}_runner/generic_{system}_runner.py. This is where you put shared logic specific to your system: engine initialization, hyper-parameter configuration, token usage extraction, and monetary cost calculation. For example, GenericLotusRunner (at src/runner/generic_lotus_runner/) initializes the LOTUS LM, configures reasoning effort per model, calls _discover_query_impl() to dispatch to _execute_q*() methods, and extracts token stats from lotus.settings.lm.stats. GenericBigQueryRunner (at src/runner/generic_bigquery_runner/) sets up the BigQuery client, calls _discover_query_text() to load .sql templates, uses Jinja2 to substitute <<variables>>, and queries inference logs for cost tracking.

4 Implement Your Queries

There are two approaches depending on your system’s mode:

Code mode — Create a per-scenario runner at src/scenario/{scenario}/runner/{system}_runner/{system}_runner.py and implement _execute_q*() methods. Each method loads data, applies semantic operators (e.g., sem_filter, sem_join, sem_map), and returns a DataFrame. See the movie LOTUS runner for examples.

SQL / Code* mode — Write standalone query files in files/{scenario}/query/{system}/. SQL queries (Q{id}.sql) are templates with <<variable>> placeholders substituted at runtime. Code* queries (Q{id}.py) are Python files with a run() function that receives the data directory. The per-scenario runner itself (at src/scenario/{scenario}/runner/{system}_runner/{system}_runner.py) is minimal — just inherit from your system’s base runner and set defaults.