Enqueuing URLs
The enqueue function schedules URLs for future scraping, passing relevant context and options to subsequent scraper stages.
Features
-
Queue URLs for Future Scraping: Adds URLs to be processed later.
-
Passes Context: Includes relevant metadata for the next scraping stage.
-
Supports Job-Level Options: Configures custom settings for the queued URLs.
Parameters
-
urls
(URL | Awaitable[URL]
): One or more URLs to enqueue. -
context
(Optional[Context]
): Additional metadata for the next scraping stage. -
options
(Optional[Options]
): Custom options for the enqueued task.
Returns
-
None: URLs are added to the scraping queue.
Usage
async def scrape(
sdk: SDK, current_url: str, context: dict[str, Any], *args: Any, **kwargs: Any
) -> None:
await sdk.enqueue("https://example.com/item/456", context={"category": "electronics"})