AI agents in analytics: beyond conversational reporting

_Conversational analytics lets you ask questions about data in everyday language—without SQL—and receive answers in everyday language too._

It sounds great in theory, but as Anthropic itself writes, “just connecting Claude to a data warehouse can create a false sense of accuracy.” When the people who created Claude say that, I wouldn't take it lightly.

Three problems we need to solve

1. Metric ambiguity: a user wants to know last month's revenue, but the data warehouse contains one revenue metric modeled for management reporting and another for financial reporting. On top of that, it isn't clear whether the user wants revenue with or without VAT, or how returns should be treated.

1. Data going stale: building the right data foundations and tuning everything is great, but the company keeps moving. Both the assets and the agent's “knowledge” need to move with it.

1. Information that can't be found: the opposite of metric ambiguity. Here, the agent isn't struggling to choose among several metrics; it can't find the information in the data at all.

How to solve these problems

The data warehouse itself

The good news is that almost all of this was already best practice for building a data warehouse long before analytical agents arrived. A well-governed, documented, and structured warehouse gives both people and agents a useful place to work. Data lakes still serve valuable purposes, but an agent needs clear structure, descriptions, and governance to navigate data reliably.

We need to model data efficiently to minimize future query costs, but above all, we need to model it clearly. The goal is a clean L3 layer that both analysts and agents can navigate.

Within the data warehouse, we also manage roles, data masking, and limits. This means restricting what the agent can access, whether for legal reasons such as personally identifiable information, because we consider the data too sensitive, or because it isn't ready and we don't want the agent needlessly crawling through it and burning tokens. The smaller the playing field we mark out for the agent, the better and cheaper its answers will be.

We almost always build data warehouses in BigQuery. The reason is simple: it is the only direct connection to Google Analytics. If a company already has a data warehouse elsewhere—Microsoft Fabric or Amazon Redshift, for example—we develop the existing one. But unless we settle for the limited data available through the GA API, we still need at least a stream from BigQuery.

The industry standard for data modeling is dbt. When we work with BigQuery, we mostly use Dataform, which is part of Google Cloud Platform. Either way, we prefer a modular architecture so that we can replace part of the infrastructure when needed.

The semantic layer

The semantic layer is one of the few overhyped data approaches of recent years that remains relevant in the agentic era. At its core, it is a deceptively simple concept: an explanatory dictionary. Don't confuse it with a data dictionary, which overlaps heavily with the semantic layer but is still slightly different. In one place, it maps analytical metric names to real business metrics, explains how they are calculated, defines their permitted breakdowns—whether marketing costs can be broken down only by source and campaign, or all the way to the product or customer level—and records what they can sensibly be joined with in our model.

We mostly build the semantic layer in cube.js. Here again, we prefer open-source projects, and our main concern is keeping individual components modular and replaceable. That helps us avoid lock-in to visualization tools, which often encourage companies to build the semantic layer directly in their environment.

The data catalog

This completes the holy trinity for preparing a data warehouse for agentic analytics. A good data catalog should tell us at least who owns a metric, whether it contains personally identifiable information, how often it is updated, and what its lineage looks like—where it comes from and how it is calculated. The catalog may look like the least important of the three, but it remains integral.

One of the largest e-commerce projects in the Czech Republic is now testing agentic analytics without a semantic layer or data catalog, relying purely on a very well-prepared data warehouse. When asked how accurate the answers had been, the team said the handful of test runs had gone very well. In only one query, the agent failed to realize it was returning data for the entire group rather than the Czech operation alone. Fortunately, that is a large enough error for almost anyone to notice. Either way, it is exactly the kind of mistake that lineage should eliminate.

For the data catalog, we use Knowledge Catalog within GCP or DataHub among the open-source options. But there are many more choices.

And the goal? A certified data product: a documented metric, how it is calculated and broken down, which dimensions it can be shown with, its basic rules and business context, who owns it, how often it is updated, whether it contains personally identifiable information, what its lineage looks like, and where it actually lives.

An overview of what the data warehouse, data catalog, and semantic layer contain—and what each means in practice.

It is tempting to have Claude Code generate all of this immediately, but Anthropic warns rather emphatically that it tried that too, with poor results. At least for now, the best approach appears to be having people prepare the data foundations—perhaps with help from AI—and only then inviting agents onto the prepared playing field.

And what about skills?

Honestly, this is the part we didn't consider especially important—at least not as important as Anthropic suggests. It says that before introducing skills into agentic analytics, its system couldn't score above 21% in its evals, which used classifiers to assess the agent's answers. As skills were gradually introduced, that score rose as high as 95%. That result is at least slightly misleading, however. One of the main checkpoints in Anthropic's evals is whether the agent queries the semantic layer first and goes elsewhere only if necessary. That is precisely one of the behaviors it couldn't enforce consistently without skills.

Of course, anyone who has played with Claude Code for more than an afternoon knows that skills are no cure-all. Sometimes the agent decides otherwise, no matter how clearly you describe the recommended procedure.

And note that when Anthropic didn't keep its skills up to date, accuracy fell from 95% to 65% within a month.

---
name: [warehouse-skill]
version: [x.y.z]
description: "IF the user asks to query [the company]'s data warehouse for any
  [list of business domains] question — THEN invoke this skill. DO NOT invoke
  for [adjacent engineering tasks] or questions with no data-warehouse component."
---

# [Warehouse] Skill Instructions

## Description
The single source of truth for safe and effective [warehouse] querying.
Referenced by other skills [listed] for query execution guidance.

Act as a Data Analyst, providing strategic insights and data-driven
recommendations but seek guidance along the way.

**Out-of-scope decisions**: [product areas, etc.] → surface data only,
state "decision is [owning team]'s call", do NOT take a position or author
code fixes.

## Executing queries
Priority:
1. **[Managed connection]** (if available): [query tool] / [schema tool]
2. **[CLI fallback]** (if installed): [default project, fallback project]
3. **Neither** — ask the user to authenticate, then stop

# Semantic Layer (REQUIRED first step)

The governed semantic layer is the **mandatory default path** for every data
question — same numbers as [the BI tool], joins/grain/filters baked in. Raw SQL
via the reference docs below is the **fallback**, used only after the
semantic-layer path is shown not to cover the ask.

## Required workflow
1. **Load** — [how to load the semantic layer in each runtime, with fallbacks]
2. **Discover** — search measures/dimensions by keyword; **always check
   segments** (the named canonical population filters — hand-rolled WHERE
   clauses for these are the dominant wrong-answer mode)
3. **Compile + run** — build the spec → compile to SQL → execute
4. **Fallback** — only if discovery finds no relevant metric or compile fails
   → raw SQL via `references/*.md` (PART 3 below)

> **Don't bail early.** Do NOT fall back to raw SQL on these grounds:
> - "[custom date filtering / cohorts]" → [covered by time-dimension specs]
> - "[needs a join]" → [the metric layer already encapsulates its joins]
> - [3–4 more pre-rebutted excuses agents use to skip the semantic layer]

### Date windows & timezone — decide before you query
- **As-of date vs trailing-N days**: [convention for each]
- **"Last week/month"** → the last *complete* calendar week/month, not trailing-7/30
- **Timezone default**: [TZ]; [exception for certain reporting rollups]
- **Freshness lag**: [some] tables settle late — anchor on MAX(date), not "yesterday"

_An example of a skill._

Why query examples may not improve accuracy

A query corpus—examples of correct SQL with results, notebooks, and historical analyses—is often suggested as preparation for agentic analytics. We have not tested its effect on answer accuracy ourselves. Anthropic reports that, in its evaluation, the corpus did not materially improve answer accuracy because the agent often could not match a new question to an existing query or analysis. That result may depend on the data, tasks, and agent setup. Analytical best practices expressed as skills can also be easier for an agent to apply than a procedure it must infer from a corpus of past work.

Is 95% really enough?

The accuracy Anthropic achieved sounds impressive. But we should remember what it represents:

1. a thoroughly prepared data warehouse, semantic layer, and data catalog, plus well-maintained skills;
2. Anthropic's internal setup, including models and limits that may not be available or suitable for every company;
3. and even then, one in every twenty answers is wrong or misleading.

Anthropic probably recognizes these limitations, which is why it says it uses agentic analytics for 95% of business questions. Presumably, that means it doesn't use agents for the most strategically important ones. In my opinion, that is the right approach. When the stakes aren't critical, I can use the agent and place some trust in its answer. For crucial questions, I need at least a follow-up check.

What about validation?

By now, you have probably gathered that letting agents loose on even the best-prepared data without initial accuracy testing and continuous validation is a bad idea. Unfortunately, it is also the most common approach. Companies usually stop preparing for data agents once they have adjusted the data warehouse, built a semantic layer, and launched a data catalog.

At an absolute minimum, stress-test the system before launch, whenever you change the model, and whenever you expand the data the agent can access—for example, when you also want it to analyze inventory data.

And what should such an offline validation look like?

Use a combination of easily evaluated questions from the past, for which we already have answers, and reference questions that test how the agent interprets terms such as “worst campaign” or “last week.” We also check whether it follows the recommended procedure—the semantic layer, then the corresponding L3, plus the necessary check in the data catalog—and the analytical best practices specified in its skills. This applies when testing an external agent rather than a built-in one such as Gemini in BigQuery.

Monitoring shouldn't stop with prepared queries in offline validation. We also need to monitor the agent's live outputs, because only those can reveal blind spots that would otherwise escape us. That is much more complicated: you need infrastructure that logs inputs, the agent's process, its final output, and any user feedback, while also evaluating database-query efficiency and looking for patterns in inaccurate answers.

In the webinar, we also covered cloud-based and self-hosted models, security, and vendor lock-in. For more on building reliable agent workflows, see our article on data alerts and verification for AI agents.