# Audit & provenance

For every result, kwery can hand you the complete trail of **where the data came from**:
which URLs were fetched, when, from which country, how long each fetch took, and what HTTP
status the marketplace returned. This is the audit backbone for MAP enforcement, compliance
work, and any situation where "we saw this price" needs to be *provable*, not just stated.

> **Rolling out.** The `include_meta` provenance block is being enabled account by account
and may not yet be attached to your results. Treat `meta` as **optional** in your pipeline
for now, and [contact us](https://kwery.co/contact) if your workflows need the crawl trail
today — we'll prioritize your account. The shape documented below is final.


## What provenance answers 

- *"Prove this price existed."* — the exact source URL plus an observation timestamp, per
fetch.
- *"When exactly was this observed?"* — `observed_at`, ISO 8601, UTC, per fetch.
- *"Was this seen from the right market?"* — `geo`, the country the request was made from
(a `.de` price observed from `de`, not through some other locale).
- *"Is this data point disputable?"* — every fetch has a stable `crawl_id`; quote it to
support and we can trace that individual fetch end to end.


## Enabling it 

Provenance is **opt-in per request**, so you only carry the extra payload when you need it:

- **Batch** (`POST /job`) — set `include_meta: true` on the request.
- **Stream** (`POST /stream`) — set `include_meta: true`; the provenance block is attached
to each result in every webhook delivery.


```bash
curl -s https://api.kwery.co/job \
  -H "Authorization: Bearer $KWERY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "idealo",
    "country": "de",
    "key": "gtin",
    "values": ["4006381333962"],
    "include_meta": true
  }'
```

There is no extra charge — provenance rides along with the result you already paid for.

## The `meta.crawls[]` trail 

With `include_meta` enabled, each result carries `meta.crawls[]` — **one entry per fetch
that contributed to the result**:

```json
"meta": {
  "crawls": [
    {
      "crawl_id": "9f1c2d3e4b5a6c7d8e9f0a1b",
      "url": "https://www.idealo.de/preisvergleich/OffersOfProduct/...",
      "http_status": 200,
      "duration_ms": 2341,
      "observed_at": "2026-07-01T12:00:00Z",
      "geo": "de",
      "job_type": "idealo.search"
    }
  ]
}
```

| Field | What it tells you |
|  --- | --- |
| `crawl_id` | Stable ID for this individual fetch — quote it to support when asking about a specific data point. |
| `url` | The exact URL that was fetched. |
| `http_status` | The HTTP status the target returned (e.g. `200`). |
| `duration_ms` | How long the fetch took, in milliseconds. |
| `observed_at` | When the data was observed (ISO 8601, UTC). |
| `geo` | The country the request was made from. |
| `job_type` | The internal crawl step that produced this fetch (e.g. `idealo.search`). |


### Why one result can list several crawls 

Most marketplaces need more than one fetch to assemble a complete answer, and every fetch
appears as its own entry:

- **A GTIN lookup** typically runs a *search* fetch (resolve the GTIN to the marketplace's
product) plus a *product/offer* fetch.
- **`search_and_offers`** adds a seller-list fetch on sources where offers live on a
separate endpoint (Amazon, Kaufland).
- **Paginated offer lists** (Idealo) add one entry per extra page fetched.
- A **cache-served** result carries the trail of the *original* crawls — `observed_at`
shows you exactly how old the observation is (the same age the
[trust `freshness` component](/guides/trust#components) scores).


Read together, the entries are the full story of how your answer was produced.

## Audit workflows 

- **MAP / pricing disputes** — store `url` + `observed_at` + `crawl_id` alongside the
price. A violation report that says *"€189.00 on idealo.de, observed 2026-07-01
12:00 UTC from Germany, crawl 9f1c…"* is evidence; a bare price is an anecdote.
- **Compliance & retention** — results (with their provenance) are downloadable for
**72 hours**; persist them on your side as part of your evidence chain. Enterprise plans
add managed **audit-trail export** — the complete provenance stream delivered to your
storage — see [Pricing → Enterprise](/guides/pricing#enterprise).
- **Debugging with support** — anything odd about a data point, send us the `crawl_id`.
It pins the conversation to one specific fetch instead of "sometime yesterday".


## Relation to Trust 

[Trust](/guides/trust) and provenance answer different questions about the same result:
**trust** scores *how much to rely on it*; **provenance** proves *where it came from*.
High-stakes pipelines typically use both: route on `trust.score`, archive `meta.crawls`
for whatever you act on.

## Questions we get 

- **Does it cost extra?** No — no extra credits. It's off by default only to keep payloads
small.
- **How much bigger are payloads?** One crawl entry is ~200 bytes; a typical result carries
one to three entries (plus one per extra Idealo offer page).
- **Can I get provenance retroactively for a past result?** Within the 72-hour retention
window, re-download the job — provenance is part of the stored result if the job was
submitted with `include_meta: true`. It cannot be reconstructed for jobs submitted
without it, so watch lists that may ever need evidence should always set it.
- **Is the raw HTML available?** Not via the API. For enterprise evidence-chain
requirements (stored page snapshots), [talk to us](https://kwery.co/contact).


## Next steps

- [Trust](/guides/trust) — the confidence score carried by every result.
- [Change events](/guides/change-events) — notifications when watched products change.
- [Webhooks](/guides/webhooks) — stream deliveries, where provenance is attached per result.