A fictional commerce business. Synthetic data. A real place to inspect how AI analytics answers are checked.
Shorelane is Nodal’s open reliability testbed, with a public warehouse, dbt models, and business questions with documented reference answers. Nodal builds and maintains it as a companion to the free Nodal Context tools.
Shorelane is a demonstration environment, not a Nodal customer or a production case study.
“What was our revenue in Q1 2024?”
A query can run successfully and still use the wrong business definition. Shorelane includes realistic ambiguity and data debt so you can inspect how an agent chooses a measure, states assumptions, and checks its result.
Shorelane also distinguishes net revenue and billed revenue. Each measure answers a different business question.
Shorelane’s resolution rule asks which measure the requester means. If an unqualified answer is required, it uses recognized revenue and states that assumption. Marketing and treasury questions can call for GMV or collected cash instead.
The metric definitions establish the meaning; the generated data and documented reference calculation establish the expected result. The question set records which interpretation to check.
Read the revenue definitions ↗Inspect the question set ↗ and the reference calculation ↗. A correct answer must match the intended measure and time window, not just return a plausible number.
The product recording demonstrates a dbt change, a replay of saved questions, and a review of the affected answers.
The demo setup changes which period refunds belong to: their refund date or the original order date. Replaying the saved queries exposes which answers move. Review those changes against approved references to decide whether the new behavior is intended or a regression.
This demonstrates a workflow in a controlled environment. It does not establish general agent accuracy or customer outcomes. The healthcare report on the product page is a separate illustrative example.
Watch the evaluation workflow →Use the repository’s maintained instructions to connect an agent, ask a question, and check the answer against its documented reference.
Keep track of the dataset version, question set, agent and model, context, and evaluation method. Results depend on that configuration. Adding context can resolve a documented ambiguity; it does not guarantee every answer is correct.
Shorelane is an independent, open reimplementation of the fictional-company concept described in Hex’s “How we evaluate data agents” ↗. See the repository for provenance and environment details.
Nodal Context helps your analyst build business context and evaluation cases together. Your team maintains the references, runs comparisons, and reviews failures.
Get the free tools ↗Local checks use a model judge to review generated SQL requirements; they do not execute warehouse queries. In-session live answer verification is a separate guided workflow with approved data access. Model and warehouse usage are separate.
Start with one domain, one warehouse, one supported agent workflow, and 10–15 questions over two weeks after access is ready. Your team keeps the context, tests, and reports, with a handoff plan.
See the evaluation sprint →One-time introductory engagement. Platform, model, and warehouse costs and custom integrations are separate.