ardvark crawl

$ ardvark seed ct --count 1000
seed ct_log complete: 1000 entries from oak,argon,nimbus log(s) · 847 added · 153 skipped (already queued)
$ ardvark crawl
  hit   acme.com            well_known   catalog valid          14 entries
  hit   tools.example.dev   robots_agentmap  valid_with_warnings queries.count
  miss  blog.someone.net    well_known   404
  hit   broken.startup.ai   well_known   invalid                urn.format ×3
  ...
run complete: 41 pages fetched · 847 hosts probed · 3 catalogs found · 2 valid · 1 error

Every probe recorded — hits, misses, and broken catalogs alike Per-check verdicts — schema + 7 semantic rules, stored per catalog Resumable — kill it anytime; the queue lives in the database

What it does

A registry-grade crawler for the ARD ecosystem, in one binary.

Finds catalogs three ways

/.well-known/ard.json (falling back to /.well-known/ai-catalog.json), Agentmap: directives in robots.txt, <link rel="ard"> tags, and inline application/ld+json catalogs on crawled pages.

Resolves everything

Recurses into nested catalogs, fetches the agent cards and MCP server cards entries point at, and harvests discovered registries — referrals included.

Verifies against spec

Official JSON Schema plus semantic checks: URN grammar, url-xor-data, query counts. Each check stored pass/fail — a report card per catalog.

Seeds itself from CT logs

ardvark seed ct pulls freshly-issued certificates from Certificate Transparency logs — new deployments are where adopters show up first.

Stores anywhere

SQLite out of the box, MySQL or Postgres with one config key. Raw documents kept verbatim, plus a JSONL event log of every discovery.

Crawls politely

Per-host rate limiting, robots.txt compliance, size and redirect caps, exponential backoff. One host having a bad day never sinks a run.

Works with AI agents

ardvark mcp exposes crawl, search, and read tools over stdio. Agents can query the index without shell commands or direct SQL. Set up MCP.

Quickstart

From zero to an indexed dataset in three commands.

Install

Homebrew, go install, or a release binary.

brew install helgesverre/tap/ardvark

Crawl

Seed with a URL, a list, or CT-log domains.

ardvark crawl https://example.com
# or probe hosts directly, no spidering:
ardvark probe huggingface.co github.com

Use the data

Search the index, inspect totals, or export every entry.

ardvark stats
ardvark search "database agent" --verdict valid
ardvark export --format jsonl --out resources.jsonl

The crawler for AI catalogs.
Discover, verify, index.