Self-hostable agents that turn the live web into spreadsheets

What it does

BigSet is a self-hostable tool that generates structured datasets from the live web. You describe what you want—say, YC companies currently hiring engineers with their funding stage and location—and an orchestrator agent infers the schema, dispatches sub-agents to search and verify facts across public pages, deduplicates the results, and delivers a browsable table you can export as CSV or XLSX. You can also set a refresh cadence so the agents re-run on schedule, keeping the dataset from going stale.

The interesting bit

The architecture treats a natural language prompt as a job spec for a tiny research team: one agent figures out the schema and where to look, then fans out parallel workers to investigate each entity and cross-check what they find against live sources. It is essentially an autonomous ETL pipeline where the transformation layer is just LLMs reading web pages and deciding whether a row is verified enough to keep.

Key highlights

• Runs locally with Docker and self-hosted Convex/Postgres, or via a cloud tier with a row-operation quota.
• Targets publicly available web data; agents search, fetch, and verify in real time, so generation takes a few minutes per dataset.
• Outputs are downloadable files, not a queryable database—SQL support is on the roadmap.
• Includes a CLI and optional curated seed datasets to demonstrate the agent workflows.
• Experimental and openly rough: schema inference is imperfect and some topics work better than others.

Caveats

• The project is explicitly experimental, with rough edges and occasional breakage.
• It only reaches public web pages; anything behind logins or paywalls is out of scope.
• Dataset generation is not instant—expect to wait two to five minutes while the agents do actual research.

Verdict

Worth a look if you regularly need niche, structured web data—competitive lists, pricing tables, hiring leads—and would rather delegate the scraping and schema design to agents than maintain another set of brittle scripts. Skip it if you need sub-minute latency, deep access to authenticated sources, or a production-grade BI backend today.