Testset generation reference

Testset generation in RAGET creates comprehensive test datasets for evaluating RAG system performance.

giskard.rag.generate_testset(knowledge_base: KnowledgeBase, num_questions: int | None = 120, question_generators: QuestionGenerator | Sequence[QuestionGenerator] | None = None, language: str | None = 'en', agent_description: str | None = 'This agent is a chatbot that answers question from users.') → QATestset[source]

Generate a testset from a knowledge base.

Parameters:

  • knowledge_base (KnowledgeBase) – The knowledge base to generate questions from.

  • num_questions (int) – The number of questions to generate. By default 120.

  • question_generators (Union[BaseQuestionModifier, Sequence[BaseQuestionModifier]]) – Question generators to use for question generation. If multiple generator are specified, num_questions will be generated with each generators. If not specified, all available question generators will be used.

  • language (str, optional) – The language to use for question generation. The default is “en” to generate questions in english.

  • agent_description (str, optional) – Description of the agent to be evaluated. This will be used in the prompt for question generation to get more fitting questions.

Returns:

The generated test set.

Return type:

QATestset

class giskard.rag.QATestset(question: Sequence[QuestionSample])[source]

A class to represent a testset for QA models.

copy()[source]

Return a copy of the testset.

count_sample_by_metadata(metadata_name: str)[source]

Count the number of samples for each value of a metadata field.

get_metadata_values(metadata_name: str)[source]

Return the unique values of a metadata field.

classmethod load(path)[source]

Load a testset from a JSONL file.

Parameters:

path (str) – The path to the input JSONL file.

classmethod load_from_hf_hub(repo_id: str, token: str = None, **kwargs: Any) → QATestset[source]

Load an instance of the class from the Hugging Face Hub.

Parameters:

  • repo_id (str) – The repository ID on the Hugging Face Hub.

  • token (str, optional) – Authentication token for private repositories. Defaults to None.

  • **kwargs (Any) – Additional arguments passed to load_dataset.

Returns:

An instance of the class itself loaded from the Hub.

Return type:

QATestset

Raises:

ImportError – If required dependencies are not installed.

push_to_hf_hub(repo_id: str, token: str = None, private: bool = False, **kwargs: Any) → CommitInfo[source]

Push the QATestset to the Hugging Face Hub.

Parameters:

  • repo_id (str) – The repository ID on the Hugging Face Hub.

  • token (str, optional) – Authentication token for private repositories. Defaults to None.

  • private (bool) – Whether to create a private repository. Defaults to False.

  • **kwargs (Any) – Additional arguments passed to Dataset.push_to_hub().

Returns:

The commit information.

Return type:

CommitInfo

save(path)[source]

Save the testset as a JSONL file.

Parameters:

path (str) – The path to the output JSONL file.

to_pandas(filters: dict | None = None)[source]

Return the testset as a pandas DataFrame.

Parameters:

filters (dict, optional) – A dictionary of filters to apply to the testset. The keys are the metadata names and the values are list of kept values. If multiple metadata are provided, the filters are combined with an AND operator.

to_test_suite(name=None, slicing_metadata: Sequence[str] | None = None)[source]

Convert the testset to a Giskard test suite.

Parameters:

name (str, optional) – The name of the test suite. If not provided, the name will be “Test suite generated from testset”.

Returns:

The test suite.

Return type:

giskard.Suite