Stream endpoints

Large-scale crawls with incremental webhook delivery. Supports up to 100 000 values per job. Results are pushed to your endpoint as they complete — no polling required.

Operations

Submit a stream job

Request

Submit a large batch for incremental webhook delivery. Results are pushed to callback_url in batches as they complete.

Webhook payload

{
  "job_id": "...",
  "client_ref": "...",
  "batch_sequence": 1,
  "is_final": false,
  "results": [ ... ]
}

Signature verification

Each delivery is signed with HMAC-SHA256. Verify using the webhook_secret returned at submission:

X-Signature: sha256=<hex digest>

Compute HMAC-SHA256(webhook_secret, raw_request_body) and compare.

Fallback

If webhook delivery fails after all retries, use POST /stream/{id}/replay-all to re-queue dead-lettered deliveries, or retrieve results directly via GET /stream/{id}/results.

Limits

Parameter	Default	Maximum
`values`	—	100 000
`delivery_batch_size`	100	200
Job timeout	1 440 min	1 440 min

Security

ApiKeyAuth or BearerAuth or BasicAuth

Bodyapplication/jsonrequired

sourcestringrequired

Enum:"idealo""amazon""google""ebay""kaufland""otto""billiger""guenstiger""walmart""lidl"

countrystringrequired

ISO 3166-1 alpha-2 country code (lowercase).

Example:"de"

keystring

Type of values being submitted. Valid values depend on source.

Default:"term"

topicstring

Processing mode. Valid values depend on source.

Default:"search"

valuesArray of strings, <= 1000 itemsrequired

Values to query. Maximum 1 000 per request. Duplicates are removed.

A comma-separated string is also accepted as a convenience form ("a,b,c" → ["a","b","c"]), but is not compatible with key: product — product values contain commas internally.

Example:

[
  "4006381333962",
  "4719512101148"
]

max_pagesinteger, [ 1 .. 10 ]

Number of result pages to fetch per value. Applicable to Idealo and Amazon. Ignored by Google and eBay.

Default:1

max_ageinteger, [ 0 .. 259200 ]

Maximum acceptable age of a result, in seconds. Default: 1200 (20 minutes). Maximum: 259200 (3 days) — the retention period for results; larger values are clamped.

When a value was already crawled inside this window, the stored result is returned without re-crawling it and the response is not charged. Set to 0 to always force a fresh crawl.

Available on eligible accounts; elsewhere every value is always crawled fresh.

Default:1200

sort_bystring

Sort order for offers within each result.

Default:"ranking_descending"

timeoutinteger, [ 1 .. 1440 ]

Job hard timeout in minutes. The job is forcibly finished with whatever partial results exist after this time. Default: 1 440 (24 h).

Default:1440

callback_urlstring or null, (uri)required

Webhook URL to receive results when the job finishes. For incremental push delivery, use the Stream endpoint instead.

not_found_max_ageinteger, >= 1

How many days a not-found cache entry is honoured before the product is re-queried. Applies to GTIN and PZN lookups (Idealo gtin/pzn, Amazon gtin, Google product). Set to 1 to force a fresh lookup on the next job after a single day. Ignored for term and id keys.

Default:10

include_metaboolean

When true, each result carries a meta provenance block — the crawl trail (meta.crawls[]) with URL, HTTP status, timing and observation timestamp per fetch. See Audit & provenance.

Default:false

waitinteger or boolean

One of:

Hold the request open for up to this many seconds (true = the maximum, 120) and, if the job finishes in time, return the results inline — the response gains a results array identical to what GET /job/{id}/download serves. If the job is still running when the wait elapses, the normal job object is returned with wait_timed_out: true; keep waiting with GET /job/{id}?wait=N or poll GET /job/{id} as usual. Accepts exactly one value (more returns "error": true); leave it off for batches.

[ 1 .. 120 ]

integer

client_refstring or null

Opaque reference string echoed in every webhook payload and in the status response. Use to correlate jobs with your own records.

Example:"batch-2026-04-03"

delivery_batch_sizeinteger, [ 1 .. 200 ]

Number of results per webhook delivery.

Default:100

fetch_seller_namesboolean

Idealo only. When true, seller detail pages are crawled for marketplace offer sellers whose names are not yet in the database. Seller names already cached in the database are always applied regardless of this flag.

Enabling this adds one additional request per unresolved seller, billed at the standard rate.

Default:false

curl -i -X POST \
  https://docs.kwery.co/_mock/api-reference/stream \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: YOUR_API_KEY_HERE' \
  -d '{
    "source": "idealo",
    "country": "de",
    "key": "gtin",
    "values": [
      "4006381333962",
      "4719512101148"
    ],
    "callback_url": "https://your-server.example.com/webhooks/prices",
    "client_ref": "batch-2026-04-03",
    "delivery_batch_size": 50
  }'

Responses

Stream job created

Bodyapplication/json

One of:

errorboolean

Example:false

job_idstring

webhook_secretstring

32-byte hex secret for verifying webhook signatures. Store securely — not retrievable after submission.

client_refstring or null

Response

{
  "error": false,
  "job_id": "64f1c2d3e4b5a6c7d8e9f0a1",
  "webhook_secret": "a3f8c2d1e9b4...",
  "client_ref": "batch-2026-04-03"
}