Question generation reference

Question generation in RAGET automatically creates test questions to evaluate RAG system performance.

class giskard.rag.question_generators.SimpleQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]

Base question generator that generates questions from a KnowledgeBase.

Parameters:

  • context_neighbors (int, optional) – Number of context neighbors to use for question generation.

  • context_similarity_threshold (float, optional) – Similarity threshold to keep neighboring document during question generation.

  • context_window_length (int, optional) – Context window length of the llm used in the llm_client of the generator.

  • llm_client (LLMClient, optional) – The LLM client to use for question generation. If not specified, a default openai client will be used.

  • llm_temperature (float, optional) – The temperature to use in the LLM for question generation. The default is 0.5.

generate_single_question(knowledge_base: KnowledgeBase, agent_description: str, language: str, seed_document=None) → dict[source]

Generate a question from a list of context documents.

Parameters:

context_documents (Sequence[Document]) – The context documents to generate the question from.

Returns:

The generated question and the metadata of the question.

Return type:

QuestionSample

class giskard.rag.question_generators.ComplexQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]

Complex question generator that generates questions from a KnowledgeBase. This generator is a subclass of the _BaseModifierGenerator class. Hence it has a _base_generator attribute that is an instance of the SimpleQuestionsGenerator class.

Generates first simple question that will be complexified.

Parameters:

  • context_neighbors (int, optional) – Number of context neighbors to use for question generation.

  • context_similarity_threshold (float, optional) – Similarity threshold to keep neighboring document during question generation.

  • context_window_length (int, optional) – Context window length of the llm used in the llm_client of the generator.

  • llm_client (LLMClient, optional) – The LLM client to use for question generation. If not specified, a default openai client will be used.

  • llm_temperature (float, optional) – The temperature to use in the LLM for question generation. The default is 0.5.

class giskard.rag.question_generators.DistractingQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]
class giskard.rag.question_generators.SituationalQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]
class giskard.rag.question_generators.DoubleQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]
class giskard.rag.question_generators.ConversationalQuestionsGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]
class giskard.rag.question_generators.OutOfScopeGenerator(context_neighbors: int = 4, context_similarity_threshold: float = 0.2, context_window_length: int = 8192, llm_client: LLMClient | None = None, llm_temperature: float = 0.4)[source]

Out of Knowledge Base question generator that generates questions from a KnowledgeBase.

Parameters:

  • context_neighbors (int, optional) – Number of context neighbors to use for question generation.

  • context_similarity_threshold (float, optional) – Similarity threshold to keep neighboring document during question generation.

  • context_window_length (int, optional) – Context window length of the llm used in the llm_client of the generator.

  • llm_client (LLMClient, optional) – The LLM client to use for question generation. If not specified, a default openai client will be used.

  • llm_temperature (float, optional) – The temperature to use in the LLM for question generation. The default is 0.5.

generate_single_question(knowledge_base: KnowledgeBase, agent_description: str, language: str, seed_document=None) → QuestionSample[source]

Generate a question from a list of context documents.

Parameters:

  • knowledge_base (KnowledgeBase) – The knowledge base to generate the question from.

  • agent_description (str) – The description of the agent to generate the question for.

  • language (str) – The language to generate the question in.

Returns:

The generated question and the metadata of the question.

Return type:

Tuple[dict, dict]