The data-prep glue Meta uses to feed documents to Llama

What it does Synthetic Data Kit is essentially glue code that orchestrates the boring but necessary journey from raw documents to structured training data. It ingests files like PDFs, HTML, and Word documents, uses an LLM to generate QA pairs or chain-of-thought traces, scores the results with a Llama-as-judge curation step, and exports everything into formats like Alpaca or HuggingFace datasets. The goal is to remove the manual scripting that usually sits between your file dump and your fine-tuning run.

The interesting bit The tool treats synthetic data as a production pipeline rather than a one-off notebook. It stores intermediate results in Lance format by default, automatically chunks large documents to stay within context windows, and can even extract images alongside text for multimodal QA. The built-in curation layer is the standout feature: instead of blindly trusting generated examples, it runs a second LLM pass to score and threshold quality before anything reaches your training set.

Key highlights

• Ingests PDFs, HTML, DOCX, PPTX, TXT, and YouTube transcripts with batch directory support
• Generates QA pairs, chain-of-thought traces, summaries, and multimodal QA from ingested content
• Curates output using a Llama-as-judge quality filter with configurable score thresholds
• Exports to Alpaca, ChatML, OpenAI fine-tuning, and HuggingFace dataset formats
• Stores parsed data in Lance format and handles oversized documents via configurable chunking with overlap

Caveats

• Directory processing can hit “Invalid format” failures on some files, as shown in the README’s own verbose output example
• Still supports a legacy directory structure alongside the current one, suggesting some evolutionary baggage
• Requires a separate LLM backend such as vLLM or an external API endpoint for all generation and curation

Verdict Useful if you have a pile of unstructured documents and want to bootstrap a fine-tuning dataset without writing custom parsers and format converters. Less compelling if your data is already clean and structured, or if you prefer tight control over every synthetic example.