Saving Scraped Data
The save_data function stores scraped data, ensuring that all details are validated and saved for further use. It is typically called when scraping detailed pages.
Features
-
Stores Scraped Data: Saves extracted information from web pages.
-
Validation Support: Ensures the data follows the expected schema (if provided).
-
Automatic URL Tagging: Attaches the source URL to each saved data entry.
-
Notifies Observers: Triggers notifications to observers handling data processing.
Parameters
-
data
(ScrapeResult
): One or more dictionaries containing the extracted data.
Returns
-
None: Data is saved and processed by observers.
Usage
async def scrape(
sdk: SDK, current_url: str, context: dict[str, Any], *args: Any, **kwargs: Any
) -> None:
# Extract data
data = {
"title": "Example Product",
"price": 49.99,
"availability": "In stock"
}
# Save the data
await sdk.save_data(data)