Detect business failures by generating synthetic tests

Generative AI agents can face an endless variety of real-world scenarios, making it impossible to manually enumerate all possible test cases. Automated, synthetic test case generation is therefore essential—especially when you lack real user conversations to import as tests. However, a major challenge is to ensure that these synthetic cases are tailored to your business context, rather than being overly generic.

By generating domain-specific synthetic tests, you can proactively identify and address these types of failures before they impact your users or business operations.

Document-Based Testing

The generate_document_based method creates test cases from your knowledge base, making it ideal for testing business logic and accuracy.

Before generating test cases, you need to create a knowledge base.

# Generate document-based test cases for business testing
business_dataset = hub.datasets.generate_document_based(
    model_id=model.id,
    knowledge_base_id=knowledge_base.id,
    dataset_name="Business Logic Tests",
    description="Test cases based on company knowledge base",
    n_questions=15, # total number of questions to generate
    topic_ids=["topic-uuid-1", "topic-uuid-2"]  # Optional: filter by specific topics
)

# Wait for the dataset to be created
business_dataset.wait_for_completion()

# List the conversations in the dataset
for conversation in business_dataset.conversations:
    print(conversation.messages[0].content)

Note

You can also use the Giskard Hub UI to generate business test cases if you prefer a visual interface.

Next steps