Machine-Readable Data Dictionaries for AI Agent Discovery

AI agents can answer questions grounded in customer-owned warehouse data without direct pipeline access when an embedded connectivity layer exposes a machine-readable data dictionary through governed interfaces. The dictionary must define what the data means, how current it is, and which access constraints apply. Reliable discovery and querying depend on this structured knowledge.

Why human-readable metadata is the wrong foundation for AI agents

An agent can fail before it locates a dataset, then fail again after finding it. Inconsistent terminology and repository-specific search interfaces force it to reformulate queries across fragmented sources. Once it reaches a dataset, it must reconstruct file semantics and operating procedures from prose, nested menus, and visual hierarchy. Finding the data does not resolve its meaning.

Scientific Data Skills: Enabling Agent-Ready Scientific Data Services at Scale documents both failures. The paper describes agents inferring structure from “incomplete descriptions” rather than receiving an authoritative, structured representation. When agents piece the structure together this way, they have to guess at schema, quality signals, and access constraints. The failures compound when a plausible dataset match becomes the basis for an answer whose meaning the agent has guessed.

What machine-readable actually means for a data dictionary

Machine readability requires four interdependent metadata strata. Technical metadata defines schema, types, cardinality, and constraints so an agent can construct syntactically valid queries. Business metadata supplies definitions, certified glossary terms, and ownership. Without that authority, “revenue” becomes whichever column resembles the requested label, rather than the measure the organization has designated. A valid query can still answer the wrong question.

Operational metadata adds freshness timestamps, update schedules, and lineage. Without it, an agent cannot tell current data from a six-week-old snapshot shown as live. Semantic metadata connects concepts through relationships, synonyms, and domain context. Without it, an agent lacks the structured context it needs to reason across business terms, even when each column has a description.

These strata also depend on usable underlying records. Structured formats such as JSONL, CSV, or Parquet, clear schemas, cleaned and deduplicated records, and context that connects related entities support machine consumption. APIs and natural language search give agents flexible ways to get that data. The dictionary must turn descriptions into formal definitions, ownership into a queryable field, and data quality into a runtime signal, so an agent can act without asking a human to reconstruct the context.

How the SciDSK model packages dataset knowledge for agent consumption

The Scientific Data Skill specification, or SciDSK, offers a tested design for how to package this knowledge. A skill combines dataset descriptions, scientific context, file organization, task-specific usage procedures, quality checks, and provenance. The data stays in its original repository. Instead of separating executable operations from dataset-specific context, the package makes operational guidance and interpretation available through the same retrievable artifact.

The SciDSK paper reports that Agent-SciDSK exceeds Agent-Raw by 9.62 percentage points on query retrieval Hit@1. The comparison tests structured packaging against raw material written for humans, so the case for machine readability rests on test results as well as preference. A unified search space reduces the need to route queries across heterogeneous repositories. Interpretation gains were smaller in the controlled cases. The SciDSK condition satisfied 23 of 24 assessment criteria, compared with 22 under the web-page condition, so the evidence supports a structural advantage without implying that packaging eliminates every interpretation error.

The Scientific Data Skill Bank publishes resources across six scientific disciplines, with ways to access packages, identify them over time, and trace them to source datasets. Enterprise dictionaries need the same connection between authoritative records, retrievable context, and operating procedures. The transferable unit is a package that lets an agent receive the knowledge needed to interpret a dataset together with the guidance needed to use it.

How MCP turns a data dictionary into a live, queryable interface for agents

Anthropic introduced MCP in late 2024 to standardize how AI applications connect to external tools and data. Unlike OpenAPI’s focus on developer tooling, MCP was designed for agent consumption, with structured tool descriptions, parameter schemas, example usage, and runtime tool discovery. For dictionary design, its contribution is delivery. It makes structured knowledge available during the interaction in which an agent needs to resolve meaning or decide whether to query.

An agent could query the business glossary for the organization’s definition of revenue and receive distinctions among gross, net, recognized, and booked revenue. It can also retrieve lineage, quality signals, and access control metadata in the same interaction. Schema inspection and read-only queries are supported in MCP’s data engineering implementations. These capabilities connect interpretation with permitted execution. A dictionary no longer has to be a static reference that the agent consults separately from the data interface.

BigQuery, Databricks, and Snowflake offer managed MCP servers, allowing organizations to use vendor-hosted endpoints and existing permission models without deploying a container. Redshift offers an official open-source MCP server through AWS Labs that runs as a local process. Hosting does not remove the freshness constraint. Warehouses refresh on batch schedules, and data that is hours old may be too old for decisions that affect money or inventory. The dictionary therefore needs to expose freshness metadata at runtime rather than leave the agent to assume that a reachable endpoint serves current data.

Where ARD fits: discovery before invocation

MCP addresses tool interaction, but an agent must first know which service to contact. The Agentic Resource Discovery specification, or ARD, addresses discovery across organizational boundaries. Google and industry partners announced it to make capabilities discoverable beyond hardcoded integrations and static endpoint lists. This is an earlier stage than runtime invocation. A well-described dictionary behind an unknown endpoint cannot help an agent searching for a relevant data service.

ARD uses catalogs and registries to close that gap. Organizations publish a machine-readable catalog at /.well-known/ard.json describing tools, APIs, skills, and agent endpoints. The v0.91 path superseded the v0.9 launch’s ai-catalog.json. Registries aggregate those catalogs so agents can search by task intent. Domain-based ownership and verification mechanisms help check that a service is genuine before a connection is made, which matters when a service an autonomous agent finds may let it trigger third-party actions.

Without a machine-readable catalog, a dictionary remains outside that catalog-based discovery path and requires a known integration. Discovery must also establish where the resource comes from. ARD’s trust mechanisms address that requirement before invocation, while MCP provides the interface through which the discovered capability becomes usable.

Agent-ready API documentation as the interface layer between agents and data systems

Fern’s March 2026 guide identifies a further failure at the API surface. Agents hallucinate parameters and produce broken code when they cannot find the context they need or parse documentation easily. Traditional HTML reference pages can consume thousands of tokens in menus, styling markup, and visual formatting. That overhead does not explain the interface. Even a discoverable, reachable dictionary can remain difficult to use if its documentation obscures the parameters and procedures the agent needs.

Fern recommends semantic search with retrieval-augmented generation to retrieve the 3–5 most relevant passages. Retrieval by intent helps agents find details in the documentation without having to rely on keyword matches or invent parameters to fill gaps. Files that use fewer tokens offer another way to get the same knowledge: llms.txt is a lightweight index of documentation pages, while llms-full.txt has the full content with HTML markup removed. Neither substitutes for accurate definitions, but both reduce the work required to locate and consume them.

One authoritative specification keeps these access paths in sync. OpenAPI specifications or equivalent sources can generate human-readable documentation, machine-readable versions, and SDK code examples without manual duplication. Production interfaces also need a clear authentication contract. OAuth 2.1 with PKCE and scoped bearer tokens supports capability restrictions, with tenant isolation for shared deployments. These are dictionary design concerns because agents interact with the exposed interface.

Security and access control as structural requirements in a machine-readable dictionary

Access constraints must be in the dictionary even when they are described in side-channel documentation or enforced at the network boundary. Agents use OAuth tokens or API keys, connect to external services, and carry out data operations faster than human users. Their credentials can permit more than a task requires. Machine-readable constraints tell agents what they need to check access before they act, while enforcement is still needed to keep them from disclosing data without permission.

The dictionary must expose who can read a dataset, under what conditions, and with which credentials. Sensitivity classifications and residency constraints must also be machine-readable before a query. Audit logging keeps a record of agent queries for governance. These controls depend on an inventory of agents and MCP connections, because an unknown connection is also an unreviewed route to data. Low-code agents deployed before security review make that dependency especially important when production access is overly permissive.

SOC 2 Type II attestation has become a procurement baseline for products exposing customer warehouses through white-label interfaces. Customers with strict rules on residency also need private-cloud or VPC options. Those options must cover region-specific data and metadata residency, customer-managed cryptographic secrets, and guardrails against cross-region deployment. A private deployment alone does not say which dataset is sensitive or where its metadata may be kept, so infrastructure controls and dictionary fields must set boundaries that agree.

What product teams building data-exporting SaaS applications must change

For product teams, the lesson is to treat the dictionary as a feature to ship with exported data. Applications that send data to customer warehouses or databases are publishing assets that agents may discover and query. The technical, business, operational, and semantic strata must remain available to agent consumers, including agents querying synced data through an embedded connectivity layer.

The delivery decision follows from that obligation. Teams need MCP for runtime dictionary queries, ARD-compatible publication through ard.json for discovery, and agent-readable documentation using llms.txt and semantic search. These components serve different stages of the workflow. Access metadata and enforced permissions then constrain what the agent can do with the data it reaches.

Building MCP servers, managing schema synchronization across 20-plus destination platforms, and handling encryption, SOC 2 compliance, and private-cloud deployment is a project measured in quarters. Embedding a white-label connectivity platform with a React SDK and a clean API can compress delivery to days. The evaluation must extend beyond connector coverage. It must include dictionary structure, runtime access, discovery, documentation synchronization, tenant isolation, and deployment controls.

Data-volume-based pricing penalizes richer metadata and more frequent synchronization, while destination-based pricing avoids penalizing data movement needed to keep dictionaries current. That makes pricing part of the freshness decision. The shipping goal is a governed interface through which an agent can understand and query customer-owned data without receiving raw pipeline access. The embedded layer is useful only when it delivers that knowledge as reliably as it delivers connectivity.