data-scraper-agent

Topics in the source

• Data Scraper Agent
• When to Activate
• Core Concepts
• The Three Layers
• Free Stack
• AI Model Fallback Chain
• Batch API Calls for Efficiency
• Untrusted Scraped Data

Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time. Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase User wants to gather or monitor any public website or API User says "build a bot that checks...", "monitor X for me", "collect data from..." User wants to track jobs, prices, news, repos, sports scores, events, listings User asks how to automate data collection without paying for hosting User wants an agent that gets smarter over time based on their decisions Every data collection agent has three layers: | Layer | Tool | Why | |---|---|---| | Scraping | requests + BeautifulSoup | No cost, covers 80% of public sites | | JS-rendered sites | playwright (free) | When HTML fetching fails | | AI enrichment | Gemini Flash via REST API | 500 req/day, 1M tokens/day — free | | Storage | Notion API | Free tier, great UI for review | | Schedule | GitHub Actions cron | Free for public repos | | Learning | JSON feedback file in repo | Zero infra, persists in git | Build agents to auto-fallback across Gemini models on quota e

This is an excerpt. Refer to the complete source and its current requirements.