Structured LLM NLP that feels like scikit-learn

What it does

Promptify exposes common NLP tasks—entity recognition, classification, question answering, summarization, and others—as Python classes backed by LLM prompts. You instantiate a task, pass a model string supported by LiteLLM, and get back Pydantic-typed objects rather than raw text. It is largely glue between prompt templates, LiteLLM, and structured-output parsing, which saves you from writing that boilerplate yourself.

The interesting bit

The library adopts scikit-learn’s interface philosophy: swap models by changing a string, call batch() for concurrent processing, or use acall() for async. The safe parser is the unglamorous but useful piece—it completes JSON for providers without native structured outputs, so you are not tied to OpenAI.

Key highlights

• Built-in tasks like NER, Classify, and QA return Pydantic models with typed fields and confidence scores.
• Supports any LiteLLM provider—OpenAI, Anthropic, Google, Ollama, Azure, and over 100 others—via a single model string.
• Includes an evaluation module with precision, recall, F1, accuracy, exact match, and ROUGE metrics.
• Offers batch processing, async execution, domain hints such as domain="medical", few-shot examples, and built-in cost tracking via get_cost_summary().

Verdict

This is for developers who want structured NLP results from LLMs without hand-rolling prompts or parsers. If you already maintain custom prompt chains and evaluation logic, it may feel like a thin convenience layer over LiteLLM.