Saving Scraped Data

The save_data function stores scraped data, ensuring that all details are validated and saved for further use. It is typically called when scraping detailed pages.

Features

  • Stores Scraped Data: Saves extracted information from web pages.

  • Validation Support: Ensures the data follows the expected schema (if provided).

  • Automatic URL Tagging: Attaches the source URL to each saved data entry.

  • Notifies Observers: Triggers notifications to observers handling data processing.

Parameters

  • data (ScrapeResult): One or more dictionaries containing the extracted data.

Returns

  • None: Data is saved and processed by observers.

Usage

async def scrape(
    sdk: SDK, current_url: str, context: dict[str, Any], *args: Any, **kwargs: Any
) -> None:
    # Extract data
    data = {
        "title": "Example Product",
        "price": 49.99,
        "availability": "In stock"
    }
    # Save the data
    await sdk.save_data(data)