Create test datasets
A dataset is a collection of conversations used to evaluate your agents. We allow manual test creation for fine-grained control, but since generative AI agents can encounter an infinite number of test cases, automated test case generation is often necessary, especially when you don’t have any test conversations to import.
This section will guide you through creating your own test datasets. In general, we cover five different ways to create datasets:
Design your own test cases using a full control over the test case creation process and explore them in the playground.
Import existing test datasets from a JSONL or CSV file, obtained from another tool, like Giskard Open Source.
Detect security failures, by generating synthetic test cases to detect security failures, like stereotypes & discrimination or prompt injection, using adversarial queries.
Detect business failures, by generating synthetic test cases to detect business failures, like hallucinations or denial to answer questions, using document-based queries and knowledge bases.