Enqueuing URLs

The enqueue function schedules URLs for future scraping, passing relevant context and options to subsequent scraper stages.

Features

  • Queue URLs for Future Scraping: Adds URLs to be processed later.

  • Passes Context: Includes relevant metadata for the next scraping stage.

  • Supports Job-Level Options: Configures custom settings for the queued URLs.

Parameters

  • urls (URL | Awaitable[URL]): One or more URLs to enqueue.

  • context (Optional[Context]): Additional metadata for the next scraping stage.

  • options (Optional[Options]): Custom options for the enqueued task.

Returns

  • None: URLs are added to the scraping queue.

Usage

async def scrape(
    sdk: SDK, current_url: str, context: dict[str, Any], *args: Any, **kwargs: Any
) -> None:
    await sdk.enqueue("https://example.com/item/456", context={"category": "electronics"})