An async-first testing kit for agents that’s still unpacking

What it does

Giskard v3 is a Python library for testing agentic systems. It wraps LLMs, black-box agents, or multi-step pipelines in async-first scenarios that check for regressions, RAG grounding, safety violations, and multi-turn coherence. The current beta package, giskard-checks, ships built-in evaluators ranging from simple string matching to LLM-as-judge checks like Groundedness and Conformity.

The interesting bit

The project is deliberately shedding the heavy dependencies of its predecessor, splitting into focused packages so you import only what you need. The README is admirably blunt that the vulnerability scanner and RAG evaluation are still either in progress or reliant on the legacy v2 codebase.

Key highlights

• Async-first Scenario API built for non-deterministic, multi-turn agent conversations
• Modular architecture where each sub-package carries only its required dependencies
• Built-in evals include regex, semantic similarity, and LLM-as-judge checks
• giskard-checks is in beta; giskard-scan and giskard-rag are marked in-progress and planned respectively
• Telemetry in giskard-core is optional and aggregated—no prompts or model outputs are transmitted, according to the docs

Caveats

• The headline features—red-teaming vulnerability scans and RAG test generation—are not yet available in v3 and still depend on the no-longer-maintained v2 branch
• Requires Python 3.12+, which may narrow adoption
• Only giskard-checks is usable today; the rest is on the roadmap

Verdict

Worth a look if you need lightweight, async regression tests for agents or RAG pipelines today, but wait for the roadmap to clear if you need automated red-teaming or synthetic dataset generation. Teams stuck on older Python versions need not apply.