ardvark documentation

A crawler and indexer for the agentic web, in one Go binary.

ardvark crawls the web for ARD (Agentic Resource Discovery) ai-catalog.json documents, verifies each one against the spec, and indexes every discovered agent, MCP server, and skill into a database. It's a single Go binary: no runtime, no external services beyond the sources it seeds from and the storage backend you point it at.

A run works in two phases. The frontier gets populated with hosts — from a URL, a domain list, or one of the seed sources below — then the crawler drains it: probing each host for a catalog, resolving what it finds (nested catalogs, referenced artifact documents, registries), and running every catalog through the verification pipeline. Everything lands in SQLite (or MySQL/Postgres) plus a JSONL event log, whether the catalog was valid, invalid, or never found at all.

Install

# Homebrew
brew install helgesverre/tap/ardvark

# Go
go install github.com/helgesverre/ardvark/cmd/ardvark@latest

Or grab a binary from the releases page — macOS, Linux, and Windows are all published.

Availability — everything on this page is in v0.3.0 or later.

Quickstart

Crawl a site and everything it links to, probing each discovered host for ARD catalogs:

ardvark crawl https://example.com

You'll see one row per probe as it happens, then a run summary — for example.com, which publishes no catalog, that's a wall of honest misses:

$ ardvark crawl https://example.com
seeded 1 of 1 requested seed(s)
  miss  example.com            well_known       404
  miss  example.com            robots_agentmap  no robots.txt
  miss  iana.org               well_known       404
  ...
run complete: 25 pages fetched · 40 hosts probed · 0 catalogs found · 0 valid · 0 errors

Or skip crawling entirely and probe hosts directly:

ardvark probe example.com huggingface.co

Seed the frontier from Certificate Transparency logs — new deployments are where adopters show up first — then work through it:

ardvark seed ct --count 1000
ardvark crawl

Or start from seeds/adopters.txt in the repository, the hosts already confirmed to publish ARD catalogs:

ardvark crawl --list seeds/adopters.txt

Results land in ardvark.db (SQLite by default) and ardvark.jsonl. See what you caught:

ardvark stats
ardvark export --format jsonl --out resources.jsonl

How discovery works

ardvark checks a host for an ARD catalog through three mechanisms. Each is attempted and recorded independently — a host publishing through more than one gets a result row for each, and every discovered catalog is fetched:

1. /.well-known/ai-catalog.json — the canonical location the spec defines.
2. Agentmap: directive in robots.txt — a pointer to a catalog served from somewhere else.
3. <link rel="ai-catalog"> tag on a crawled page — found while spidering during ardvark crawl (not tried by ardvark probe, which does no HTML spidering).

Every probe is recorded regardless of outcome, one row per attempt:

• hit — a document was found at that location. It still goes through verification; a hit can be a valid catalog, one with warnings, or an invalid one — invalid catalogs are stored too, since "found but broken" is useful data.
• miss — the location returned 404 (or equivalent); nothing published there.
• err — the request itself failed (timeout, connection error, malformed response) before a hit/miss could be determined.

A hit that resolves to a valid catalog is then recursed into: nested catalogs are followed up to ard.maxCatalogDepth, entries pointing at artifact documents (agent cards, MCP server cards) are fetched, and entries pointing at registries are harvested via POST /search, including registry referrals, up to registry.maxReferralDepth.

Verification & verdicts

Every fetched catalog is run through the same pipeline ardvark verify exposes standalone: official JSON Schema validation (Draft 2020-12) against the vendored ARD spec schema, followed by seven semantic checks the schema can't express. Each check is recorded pass/fail with a message — a report card per catalog, stored in verification_checks.

The rolled-up verdict is one of:

• valid — every check passed.
• valid_with_warnings — all error-severity checks passed; one or more warning-severity checks failed.
• invalid — at least one error-severity check failed (including schema validation itself).

Identifiers use the URN grammar urn:<nid>:<publisher>:<namespace...>:<name>. air is the namespace identifier (NID) in the ARD spec draft; ai also appears on catalogs already published in the wild, so both urn:air: and urn:ai: are accepted.

Run it standalone against anything, local or remote:

ardvark verify ./my-catalog.json
ardvark verify https://example.com/.well-known/ai-catalog.json

What failure looks like

A failing report renders ✗ for failed error-severity checks and ! for failed warnings. Here's a catalog whose entry identifier isn't a URN and whose media type is made up — note that semantic checks on a field the schema already rejected are skipped rather than double-reported:

$ ardvark verify ./broken.json
./broken.json
  ✗ schema.validation          support-agent — at /entries/0/identifier: validation failed
  ✓ identifier.unique          all entry identifiers are unique
  ! queries.count              support-agent — 0 representativeQueries present (recommended: 2-5)
  ! entry.media_type           support-agent — type "application/x-agent" is not a recognized ARD media type (spec does not enforce this strictly)
verdict: invalid
Error: verify: ./broken.json is invalid

An invalid verdict makes verify exit 1 — see exit codes — so it slots straight into CI. Publishers: the Publishing a catalog guide walks through the failures seen most often in the wild.

Seeding strategies

Seven sources fill the frontier, each trading off precision (how likely a domain is to actually publish a catalog) against breadth (how many domains it surfaces). Every source accepts --json and reads its defaults from config; seeding only queues domains — run ardvark crawl afterwards to drain the frontier.

$ ardvark seed ct --count 40
seed ct_log complete: 40 entries from oak,argon,nimbus log(s) · 40 added · 0 skipped (already queued)

Publishing a catalog

The other side of the crawler: this guide is for sites that want to be found. Publishing an ARD catalog takes one JSON file and one of three exposure mechanisms — and ardvark verify tells you before you deploy whether crawlers will accept it.

A minimal valid catalog

This is the smallest catalog that passes every check, warnings included:

{
  "specVersion": "1.0",
  "host": { "displayName": "Example Inc" },
  "entries": [
    {
      "identifier": "urn:air:example.com:agent:support",
      "displayName": "Support Agent",
      "description": "Answers questions about Example Inc products.",
      "type": "application/a2a-agent-card+json",
      "url": "https://example.com/.well-known/agent-card.json",
      "representativeQueries": [
        "how do I reset my password",
        "cancel my Example subscription"
      ]
    }
  ]
}

The moving parts: specVersion must be exactly "1.0"; each entry needs an ARD URN identifier, a displayName, a media type, exactly one of url or data, and ideally 2–5 representativeQueries (fewer only warns, but indexers use them).

Exposing it

Any one of the three discovery mechanisms is enough; publishing through several gets you found by more crawler configurations:

1. Serve it at /.well-known/ai-catalog.json — the canonical spec location, checked by every probe.
2. Add an Agentmap: directive to robots.txt, pointing wherever the catalog actually lives:

   Agentmap: https://example.com/.well-known/ai-catalog.json

   copy
3. Add a link tag to your pages — only found by spidering crawls, so don't make it your only mechanism:

   <link rel="ai-catalog" href="/.well-known/ai-catalog.json">

   copy

Verify before you publish

Run the file through ardvark verify locally, and again against the live URL after deploying. It exits 1 on an invalid verdict, so it works as a CI gate:

ardvark verify ./ai-catalog.json
ardvark verify https://example.com/.well-known/ai-catalog.json

The failures seen most often on real published catalogs:

• schema.validation on unexpected fields — the schema is strict where you'd least expect it: unknown keys at the catalog root or inside host are rejected (additional properties 'generator' not allowed), and metadata values must be scalars — a nested object fails with at /entries/0/metadata/nested: validation failed. Put custom extensions in metadata as flat string/number/boolean values.
• Unrecognized media types only warn — an unknown entry type yields a ! from entry.media_type, not an ✗; the catalog can still roll up valid_with_warnings. Prefer the recognized types (e.g. application/a2a-agent-card+json, application/mcp-server-card+json, application/ai-skill+md).
• urn.publisher_matches mismatch — a warning when the URN's publisher segment doesn't share a registrable domain with the serving host. Legitimate for aggregators publishing on behalf of others; for your own resources, make the publisher segment your own domain.

A worked example: ardvark.no

This site eats its own dog food — it publishes a catalog at /.well-known/ai-catalog.json and an Agentmap: directive in robots.txt, and the catalog verifies clean:

$ ardvark verify https://ardvark.no/.well-known/ai-catalog.json
https://ardvark.no/.well-known/ai-catalog.json
  ✓ identifier.unique          all entry identifiers are unique
  ✓ catalog.spec_version       specVersion is "1.0"
  ✓ entry.value_or_reference   urn:air:ardvark.no:skill:ardvark-cli — exactly one of url or data is present
  ✓ urn.format                 urn:air:ardvark.no:skill:ardvark-cli — identifier is a well-formed ARD URN
  ✓ urn.publisher_matches      urn:air:ardvark.no:skill:ardvark-cli — URN publisher "ardvark.no" matches serving domain "ardvark.no"
  ✓ queries.count              urn:air:ardvark.no:skill:ardvark-cli — 4 representativeQueries present (recommended: 2-5)
  ✓ entry.media_type           urn:air:ardvark.no:skill:ardvark-cli — type "application/ai-skill+md" is a recognized ARD media type
  ✓ entry.value_or_reference   urn:air:ardvark.no:tool:ardvark — exactly one of url or data is present
  ✓ urn.format                 urn:air:ardvark.no:tool:ardvark — identifier is a well-formed ARD URN
  ✓ urn.publisher_matches      urn:air:ardvark.no:tool:ardvark — URN publisher "ardvark.no" matches serving domain "ardvark.no"
  ✓ queries.count              urn:air:ardvark.no:tool:ardvark — 3 representativeQueries present (recommended: 2-5)
  ✓ entry.media_type           urn:air:ardvark.no:tool:ardvark — type "application/ai-skill" is a recognized ARD media type
  ✓ entry.value_or_reference   urn:air:ardvark.no:mcp:ardvark — exactly one of url or data is present
  ✓ urn.format                 urn:air:ardvark.no:mcp:ardvark — identifier is a well-formed ARD URN
  ✓ urn.publisher_matches      urn:air:ardvark.no:mcp:ardvark — URN publisher "ardvark.no" matches serving domain "ardvark.no"
  ✓ queries.count              urn:air:ardvark.no:mcp:ardvark — 3 representativeQueries present (recommended: 2-5)
  ✓ entry.media_type           urn:air:ardvark.no:mcp:ardvark — type "application/mcp-server-card+json" is a recognized ARD media type
verdict: valid

Three entries — a SKILLS.md skill, a docs page, and an MCP server card — each with a matching publisher URN and 2–5 representative queries. Steal the pattern.

Programmatic use

Three machine-readable surfaces, one data model: --json for scripts, the MCP server for agents, and the JSONL event log for anything that tails files. All of them expose the same structures.

$ ardvark probe example.com --json
{
  "hosts": [
    {
      "host": "example.com",
      "results": [
        {
          "method": "well_known",
          "url": "https://example.com/.well-known/ai-catalog.json",
          "http_status": 404,
          "outcome": "miss"
        },
        {
          "method": "robots_agentmap",
          "url": "https://example.com/robots.txt",
          "outcome": "miss",
          "error_detail": "no robots.txt"
        }
      ]
    }
  ],
  "summary": {
    "hits": 0,
    "misses": 2,
    "errors": 0
  }
}

ardvark mcp [flags]

{
  "mcpServers": {
    "ardvark": {
      "command": "ardvark",
      "args": ["mcp", "--config", "/path/to/ardvark.json"]
    }
  }
}

{"time":"2026-07-15T22:55:16.360059+02:00","level":"INFO","msg":"crawler: host probed","host":"ardvark.no","domain_id":1}
{"time":"2026-07-15T22:55:17.360293+02:00","level":"INFO","msg":"crawler: catalog verified","url":"https://ardvark.no/.well-known/ai-catalog.json","host":"ardvark.no","verdict":"valid","entries":3}

Querying the dataset

The crawl output is a plain relational schema (nine tables), identical across SQLite, MySQL, and Postgres — the queries below run unchanged on any of them; the sqlite3 invocations just match the default backend. Column names are snake_case versions of the model fields.

Catalogs by verdict, with the host that served them:

sqlite3 ardvark.db "
  SELECT d.host, c.verification_status, c.source_url
  FROM catalogs c
  JOIN domains d ON d.id = c.domain_id
  ORDER BY c.verification_status, d.host;
"

Entries by media type — what kinds of resources the ecosystem is actually publishing:

sqlite3 ardvark.db "
  SELECT media_type, COUNT(*) AS n
  FROM catalog_entries
  GROUP BY media_type
  ORDER BY n DESC;
"

Everything one publisher has put out, wherever it was discovered (URN segments are split into their own columns):

sqlite3 ardvark.db "
  SELECT urn, display_name, media_type, ref_url
  FROM catalog_entries
  WHERE urn_publisher = 'ardvark.no';
"

Artifacts by content hash — the same agent card served from multiple entries or hosts dedupes to one hash:

sqlite3 ardvark.db "
  SELECT content_hash, COUNT(*) AS copies, MIN(url) AS example_url
  FROM artifacts
  GROUP BY content_hash
  HAVING COUNT(*) > 1;
"

Two details worth knowing:

• artifacts.raw_body is a byte column (BLOB in SQLite) because artifacts can be binary — skill archives, gzip tarballs. SQLite's length() on a BLOB returns bytes directly, so length(raw_body) is fine; dumps made back when it was a TEXT column needed a CAST(raw_body AS BLOB) to count bytes instead of characters.
• JSON-in-text columns — catalog_entries.tags, .capabilities, .representative_queries, and .trust_manifest hold JSON text; unpack them with json_each() in SQLite (equivalents exist in MySQL and Postgres).

Commands

Every command accepts the global flags --config <path> (default: ./ardvark.json, then ~/.config/ardvark/ardvark.json on Unix/macOS or %AppData%\ardvark\ardvark.json on Windows) and --color auto|always|never. crawl, work, probe, every seed source, verify, stats, and info also accept --json — see JSON output.

ardvark crawl [url|domain]... [flags]

$ ardvark crawl https://acme.example
seeded 1 of 1 requested seed(s)
  hit   acme.example           well_known       catalog valid          14 entries
  hit   docs.acme.example      link_tag         valid_with_warnings    queries.count
  miss  static.acme.example    well_known       404
  miss  static.acme.example    robots_agentmap  no Agentmap directive
run complete: 12 pages fetched · 3 hosts probed · 2 catalogs found · 2 valid · 0 errors

ardvark work [flags]

ardvark probe <host>... [flags]

$ ardvark probe huggingface.co github.com
  hit   huggingface.co         well_known       found                  200 application/json; charset=utf-8
  miss  huggingface.co         robots_agentmap  no Agentmap directive
  miss  github.com             well_known       404
  miss  github.com             robots_agentmap  no Agentmap directive
probe complete: 1 hit · 3 misses · 0 errors

ardvark seed [ct|crtsh|tranco|github|mcp|curated|commoncrawl] [flags]

ardvark verify <path|url> [flags]

$ ardvark verify ./my-catalog.json
./my-catalog.json
  ✓ identifier.unique          all entry identifiers are unique
  ✓ catalog.spec_version       specVersion is "1.0"
  ✓ entry.value_or_reference   urn:air:example.com:agent:support — exactly one of url or data is present
  ✓ urn.format                 urn:air:example.com:agent:support — identifier is a well-formed ARD URN
  ! queries.count              urn:air:example.com:agent:support — 0 representativeQueries present (recommended: 2-5)
  ✓ entry.media_type           urn:air:example.com:agent:support — type "application/a2a-agent-card+json" is a recognized ARD media type
verdict: valid_with_warnings

ardvark export [flags]

ardvark export --format jsonl --out resources.jsonl

ardvark stats [flags]

$ ardvark stats
domains
  total                        847
    found_valid                3
    not_found                  798
    unprobed                   46
catalogs
  total                        3
    valid                      2
    valid_with_warnings        1
entries
  total                        41
    application/mcp-server-card+json 18
    application/a2a-agent-card+json 15
    application/ai-skill+md    8

ardvark info [flags]

$ ardvark info
ardvark
  version                      0.3.0
config
  path                         /Users/you/ardvark.json
  exists                       false
storage
  driver                       sqlite
  dsn                          ardvark.db
  path                         /Users/you/ardvark.db
  exists                       true
  size                         139264
log
  file                         /Users/you/ardvark.jsonl
  level                        info

ardvark migrate [flags]

ardvark migrate

ardvark mcp [flags]

Configuration

ardvark runs with sensible defaults and no config file. To change anything, drop an ardvark.json in the working directory or in your user config dir (~/.config/ardvark/ardvark.json on Unix/macOS, %AppData%\ardvark\ardvark.json on Windows), or pass --config path — explicit flag wins, then working directory, then user config dir. Relative file paths in a config file (storage.dsn for sqlite, log.file) resolve against the config file's directory, not the process's working directory — a user-dir config keeps its database and event log in ~/.config/ardvark/ unless it says otherwise. With no config file at all, the defaults land in the working directory. The file is schema-validated — a typo'd key or bad value gets a precise config.<path>: ... error, not silent misbehavior. Values present in the file override defaults; missing keys keep their default.

Environment — GITHUB_TOKEN is the only environment variable ardvark reads for its own behavior; it's required by seed github and nothing else. (The standard NO_COLOR/TERM conventions are honored for color, alongside --color.)

Storage

Everything flows from seeders into one persistent queue, out through five handler types, and into the tables — plus a JSONL event log alongside:

seed ct / crtsh / tranco / github / mcp / curated / commoncrawl    crawl url · domain · --list
                         │                                                    │
                         └───────────────────┬────────────────────────────────┘
                                             ▼
                         frontier_items — persistent queue in the DB
                         (pending → in_flight → done / failed)
                                             │
                         dispatcher + workers (crawler.concurrency)
                                             │
       ┌──────────────┬──────────────────────┼─────────────────────┬─────────────────────┐
       ▼              ▼                      ▼                     ▼                     ▼
  page_fetch     host_probe           catalog_fetch       artifact_fetch       registry_harvest
  spider HTML,   well-known +         fetch, verify,      agent cards,         POST /search,
  link tags      robots Agentmap      split entries       MCP server cards     follow referrals
       │              │                      │                     │                     │
       │              ▼                      ▼                     ▼                     ▼
       │       domains · probes    catalogs · catalog_entries   artifacts           registries
       │                           · verification_checks
       └── new hosts, catalogs, artifacts, and registries feed back into the frontier ───┘

  every step also appends an event line to the JSONL log (log.file)

Every raw document is kept verbatim alongside the extracted data, so you can re-process without re-crawling. Nine tables:

Swapping storage backends is one config key, storage.driver, plus a matching storage.dsn:

// sqlite (default) — dsn is a file path
{
  "storage": {
    "driver": "sqlite",
    "dsn": "ardvark.db"
  }
}

// mysql
{
  "storage": {
    "driver": "mysql",
    "dsn": "user:pass@tcp(127.0.0.1:3306)/ardvark?parseTime=true"
  }
}

// postgres
{
  "storage": {
    "driver": "postgres",
    "dsn": "postgres://user:pass@127.0.0.1:5432/ardvark?sslmode=disable"
  }
}

ardvark migrate creates or updates the schema for whichever driver is configured; crawl, verify, and every other command that touches the database run the same migration on open. The schema is identical across all three drivers — see Querying the dataset for working with it directly.

Distributed crawling

One frontier, many workers. When the frontier lives in MySQL or Postgres, several ardvark processes — on one machine or many — can drain it cooperatively. Each worker owns a fixed share of hosts, so exactly one worker ever talks to a given host: per-host politeness (crawler.perHostRequestsPerSecond, robots.txt) stays correct with no cross-process coordination. SQLite is single-process only — distributed crawling needs a shared MySQL/Postgres database.

Hosts are partitioned by hash across crawler.worker.count shards; each worker claims the shard matching its crawler.worker.index (0-based, less than count). Configure the count once in the shared ardvark.json and give each process its index with the --worker i/n flag. Seed the frontier, then run one worker per shard:

// shared ardvark.json — every worker points at the same database
{
  "storage": {
    "driver": "postgres",
    "dsn": "postgres://user:pass@db:5432/ardvark?sslmode=disable"
  },
  "crawler": { "worker": { "count": 4 } }
}

# seed the shared frontier once...
ardvark seed ct --count 5000
# ...then start one worker per shard (each on its own machine or core)
ardvark work --worker 0/4
ardvark work --worker 1/4
ardvark work --worker 2/4
ardvark work --worker 3/4

The --worker i/n flag overrides crawler.worker from config, so a single shared config plus a per-process flag is enough; you don't need a different config file per worker. A seeded ardvark crawl can stand in for one of the workers — it seeds every shard but only dequeues its own — so a common pattern is one crawl that seeds and works shard 0, paired with work peers for the rest.

Leases and reclaim

When a worker dequeues an item it takes a lease on it for crawler.leaseSeconds (default 600s). A worker that dies mid-item can't requeue the item itself, so once the lease expires a peer reclaims it back to pending and finishes it. The default outlasts the slowest legitimate handler (registry harvest pagination plus retry backoff), so a live worker's item is never reclaimed out from under it — raise crawler.leaseSeconds if you tune handlers to run longer, don't lower it below the slowest one.

Termination is global: each worker keeps polling until the whole shared frontier is empty, not just its own shard, then exits. Per-domain page budgets (crawler.maxPagesPerDomain) are enforced in the database, so they hold across the fleet rather than per process.

Dead workers and clocks

• Sharding is static — a host belongs to a shard, and a shard belongs to a fixed (index, count) partition. If a worker dies permanently, its shard's not-yet-started items are never leased, so lease-expiry reclaim never touches them, and its peers keep polling forever because the global pending count stays non-zero. Recovery is operational, not automatic: restart the missing worker with the same --worker index/count, or restart the whole fleet with a new count.
• Keep clocks in sync — lease expiry compares timestamps written by different machines, so run the workers' hosts under NTP. A badly skewed clock can reclaim a live worker's items early or leave a dead worker's items stranded too long.

Operational notes

• Politeness by default — crawler.perHostRequestsPerSecond rate-limits requests per host (default 1/s), crawler.respectRobotsTxt honors disallow rules (default on), and crawler.maxBodyBytes/redirect caps bound what a single response can cost. One host having a bad day doesn't sink a run.
• Resumable runs — the crawl frontier lives in the database as frontier_items, not in process memory. Kill ardvark crawl at any point and run it again: pending and in-flight items are picked back up automatically, no separate resume flag needed.
• --force — hosts probed within crawler.refreshAfterHours (default 168h / 7 days) are skipped on a normal run. --force bypasses that freshness window and re-probes them anyway.
• refreshAfterHours — tune this to control how often previously-probed hosts get re-checked; lower it for fast-moving domain sets, raise it to avoid re-hitting a stable dataset.
• Exit codes — 0 on success, non-zero on error; verify also exits 1 on an invalid verdict. Full table under Programmatic use.

Troubleshooting

• seed crtsh returns 503 or times out — crt.sh is a shared public service and rate-limits aggressively under load. Retry later, lower --count, or use --match to narrow the query; seed ct reads the CT logs directly and doesn't share this bottleneck.
• GITHUB_TOKEN is not set — seed github fails with GitHub's code search API requires authentication; set it to a personal access token. Export a token (no special scopes needed for public code search) and re-run.
• A host shows not_found but you know it publishes — the catalog probably appeared after the last probe, and hosts probed within crawler.refreshAfterHours (default 7 days) are skipped on subsequent runs. Re-probe it now with ardvark crawl <host> --force.
• Ctrl-C mid-crawl — safe. On interrupt, in-flight frontier items are requeued as pending; if a run dies without cleanup (crash, kill -9), items stranded in_flight are reclaimed at the next startup. Either way, re-running ardvark crawl resumes where it stopped.