Turning chaotic bank PDFs into structured data with YOLO and local LLMs

What it does

This project ingests unstructured bank statement PDFs and converts them into structured financial data using a custom YOLOv8 layout detection model, OCR, and LLM-powered table extraction. It then runs the data through a RAG pipeline with vector stores like Chroma and Faiss, and dispatches AG2 autonomous agents to categorize income and expenses, detect trends, and generate monthly or yearly summaries. A Streamlit interface (apps.py) provides the frontend.

The interesting bit

The pipeline treats document layout as a computer vision problem first—using a dedicated YOLO model to find tables and text regions before OCR and language models ever touch the text. It can also keep everything local via Ollama (Llama 3, Gemma 2) instead of cloud LLMs, though Google Gemini is supported too.

Key highlights

• Custom YOLOv8 model for PDF layout detection, paired with PyMuPDF and pytesseract for OCR
• RAG pipeline built on LangChain with Chroma and Faiss vector stores
• AG2 agent framework for autonomous financial analysis and report generation
• DeepEval integration for testing RAG output quality
• Streamlit web UI and Jupyter notebooks for development and experimentation

Caveats

• Much of the code lives in Jupyter notebooks, and the README explicitly lists a production-ready pipeline and Docker deployment as future work.
• The README lists a February 2026 migration date for the AG2 framework, which suggests the documentation timeline may not be current.

Verdict

Worth a look if you are building document parsing pipelines or want to experiment with agent-driven financial analysis on local hardware. Skip it if you need a finished, deployable product today—the Docker packaging and API deployment are still listed as future work.