The agentic web is publishing catalogs.
Somebody has to dig them up.

Sites are starting to advertise their AI agents, MCP servers, and skills in /.well-known/ai-catalog.json. ardvark crawls for those catalogs, verifies them against the ARD spec, and indexes every resource into a database you can build on.

$ brew install helgesverre/tap/ardvark

macOS · Linux · Windows — single binary, no runtime

Get started Source
ardvark mascot — a line-drawn aardvark
ardvark crawl
$ ardvark seed ct --count 1000
seeded 847 unique domains from oak.ct.letsencrypt.org (153 duplicates skipped)
$ ardvark crawl
  hit   acme.com            well-known    catalog valid          14 entries
  hit   tools.example.dev   robots_agentmap  valid_with_warnings queries.count
  miss  blog.someone.net    well-known    404
  hit   broken.startup.ai   well-known    invalid                urn.format ×3
  ...
run complete: 847 hosts probed · 3 catalogs · 2 registries harvested · 41 resources indexed
Every probe recorded — hits, misses, and broken catalogs alike Per-check verdicts — schema + 7 semantic rules, stored per catalog Resumable — kill it anytime; the queue lives in the database

What it does

A registry-grade crawler for the ARD ecosystem, in one binary.

01

Finds catalogs three ways

/.well-known/ai-catalog.json, Agentmap: directives in robots.txt, and <link rel="ai-catalog"> tags on crawled pages.

02

Resolves everything

Recurses into nested catalogs, fetches the agent cards and MCP server cards entries point at, and harvests discovered registries — referrals included.

03

Verifies against spec

Official JSON Schema plus semantic checks: URN grammar, url-xor-data, query counts. Each check stored pass/fail — a report card per catalog.

04

Seeds itself from CT logs

ardvark seed ct pulls freshly-issued certificates from Certificate Transparency logs — new deployments are where adopters show up first.

05

Stores anywhere

SQLite out of the box, MySQL or Postgres with one config key. Raw documents kept verbatim, plus a JSONL event log of every discovery.

06

Crawls politely

Per-host rate limiting, robots.txt compliance, size and redirect caps, exponential backoff. One host having a bad day never sinks a run.

Quickstart

From zero to an indexed dataset in three commands.

1

Install

Homebrew, go install, or a release binary.

brew install helgesverre/tap/ardvark
2

Crawl

Seed with a URL, a list, or CT-log domains.

ardvark crawl https://example.com
# or probe hosts directly, no spidering:
ardvark probe huggingface.co github.com
3

Use the data

Everything lands in SQLite + JSONL, ready to query.

ardvark stats
ardvark export --format jsonl --out resources.jsonl

Start digging.

ardvark on GitHub Read the ARD spec