Usage Recommendations #
Prioritize Default HTML Output
If you only need the target page’s HTML, simply pass the target_url. The API returns HTML by default when the format parameter is omitted.
{
"target_url": "https://example.com"
}
HTML output integrates better with existing crawler processing pipelines and is generally faster than requests requiring screenshots.
Use js_render Cautiously
Setting js_render=1 enables page rendering, which typically results in longer response times. It is recommended to test without this parameter (or set it to 0) first; enable rendering only if page content is missing.
Log the request_id
It is recommended to log the X-Request-Id response header for every request. This ID helps quickly pinpoint issues when troubleshooting request failures, timeouts, or unexpected results.
Client-Side Retries
You can implement limited client-side retries for errors such as timeouts, temporary service unavailability, or rate limiting.
Recommendations:
- Use exponential backoff.
- Do not retry indefinitely.
- Do not automatically retry errors such as invalid parameters, authentication failures, or insufficient funds.
- For rate-limiting errors, reduce concurrency or wait before retrying.
