Outsourcing close reading to an LLM, page by page

What it does read_books.py is a straightforward Python utility that walks through a PDF page by page, sends each page’s text to an OpenAI model, and extracts structured knowledge points. It maintains a growing JSON knowledge base across the entire book and periodically writes Markdown summaries at configurable intervals, plus a final recap. The script also filters out pages it considers boilerplate—tables of contents and indexes—so the API budget (hopefully) goes to actual content.

The interesting bit The pipeline is stubbornly sequential: it feeds the accumulated knowledge from previous pages into each new API call, trying to preserve narrative context rather than shredding the book into disconnected chunks. It can also resume from an existing JSON file, which is useful if the process crashes halfway through a long book.

Key highlights

• Processes PDFs sequentially with a running, page-to-page knowledge base
• Generates interval summaries (e.g., every n pages) and a final summary in Markdown
• Resume capability: loads existing JSON knowledge base to pick up where it left off
• Skips boilerplate pages (TOC, index) via content filtering
• Configuration is bare-bones: edit constants like PDF_NAME and ANALYSIS_INTERVAL directly in the script

Caveats

• Tightly coupled to the OpenAI API; switching providers means refactoring the client code
• All configuration happens by editing constants inside read_books.py—no CLI flags or external config file
• The README never discusses API pricing, rate limits, or token budgets, which seems relevant for a book-length ingestion tool

Verdict Useful if you want a simple, hackable pipeline to turn long PDFs into structured notes without building orchestration yourself. Look elsewhere if you need multi-model support, granular CLI control, or production-grade error handling.