Resources reference

Resources provide access to different entities in Giskard Hub through the client, such as projects, datasets, and models.

class giskard_hub.resources.ChatTestCasesResource(client: SyncClient)

Bases: APIResource

create(*, dataset_id: str, messages: List[ChatMessage], demo_output: ChatMessageWithMetadata | None = None, tags: List[str] | None = None, checks: List[CheckConfig] | None = None)

delete(chat_test_case_id: str | List[str]) → None

list(dataset_id: str) → List[ChatTestCase]

retrieve(chat_test_case_id: str)

update(chat_test_case_id: str, *, dataset_id: str = NOT_GIVEN, messages: List[ChatMessage] = NOT_GIVEN, demo_output: ChatMessageWithMetadata | None = NOT_GIVEN, tags: List[str] | None = NOT_GIVEN, checks: List[CheckConfig] | None = NOT_GIVEN) → ChatTestCase

class giskard_hub.resources.ConversationsResource(client: SyncClient)

Bases: APIResource

create(*, dataset_id: str, messages: List[ChatMessage], demo_output: ChatMessageWithMetadata | None = None, tags: List[str] | None = None, checks: List[CheckConfig] | None = None)

delete(conversation_id: str | List[str]) → None

list(dataset_id: str) → List[Conversation]

retrieve(conversation_id: str)

update(conversation_id: str, *, dataset_id: str = NOT_GIVEN, messages: List[ChatMessage] = NOT_GIVEN, demo_output: ChatMessageWithMetadata | None = NOT_GIVEN, tags: List[str] | None = NOT_GIVEN, checks: List[CheckConfig] | None = NOT_GIVEN) → Conversation

class giskard_hub.resources.DatasetsResource(client: SyncClient)

Bases: APIResource

create(*, name: str, description: str, project_id: str) → giskard.Dataset

delete(dataset_id: str | List[str]) → None

generate_adversarial(*, model_id: str, dataset_name: str = 'Generated Dataset', description: str = '', categories: List[str] | Any = NOT_GIVEN, n_examples: int = 10) → giskard.Dataset

Generate a dataset using the specified model and parameters.

Parameters:

• model_id (str) – The ID of the model to use for generation.

• dataset_name (str, optional) – Name of the generated dataset.

• description (str, optional) – Description of the dataset.

• categories (list, optional) – List of issue categories, each as a dict with ‘id’, ‘name’, ‘desc’.

• n_examples (int, optional) – Number of examples to generate in total, regardless of the number of categories.

Returns:

The generated dataset object.

Return type:

Dataset

generate_document_based(*, model_id: str, knowledge_base_id: str, dataset_name: str = 'Generated Dataset', description: str = '', n_questions: int = 10, topic_ids: List[UUID] | None = None) → giskard.Dataset

Generate a dataset from a knowledge base.

Parameters:

• model_id (str) – The ID of the model to use for generation.

• knowledge_base_id (str) – The ID of the knowledge base.

• dataset_name (str, optional) – Name of the generated dataset.

• description (str, optional) – Description of the dataset.

• n_questions (int, optional) – Number of questions to generate in total, regardless of the number of topics.

• topic_ids (list[UUID], optional) – List of topic IDs to filter.

Returns:

The generated dataset object.

Return type:

Dataset

list(project_id: str) → List[giskard.Dataset]

retrieve(dataset_id: str)

update(dataset_id: str, *, name: str = NOT_GIVEN, description: str = NOT_GIVEN, project_id: str = NOT_GIVEN) → giskard.Dataset

class giskard_hub.resources.EvaluationsResource(client: SyncClient)

Bases: APIResource

create(*, model_id: str, dataset_id: str, tags: List[str] = NOT_GIVEN, name: str = NOT_GIVEN, run_count: int = 1)

create_local(*, model: Model, dataset_id: str, tags: List[str] = NOT_GIVEN, name: str = NOT_GIVEN)

delete(execution_id: str | List[str])

list(project_id: str)

list_entries(run_id: str)

retrieve(run_id: str)

update_entry(run_id: str, entry_id: str, *, model_output: ModelOutput = NOT_GIVEN, results: List[EvaluatorResult] = NOT_GIVEN)

class giskard_hub.resources.KnowledgeBasesResource(client: SyncClient)

Bases: APIResource

Resource for managing knowledge bases.

create(*, project_id: str, name: str, data: str | List[dict[str, str]], description: str | None = None, document_column: str | NotGiven = NOT_GIVEN, topic_column: str | NotGiven = NOT_GIVEN) → KnowledgeBase

Create a new knowledge base.

Parameters:

• project_id (str) – The project ID.

• name (str) – The name of the knowledge base.

• data (str or list[dict[str, str]]) – Either a filepath (str) to a JSON or JSONL file, or a list of dicts containing “text” and “topic” (optional) keys.

• description (str, optional) – Description of the knowledge base.

• document_column (str, optional) – Name of the document column.

• topic_column (str, optional) – Name of the topic column.

Returns:

The created knowledge base object.

Return type:

KnowledgeBase

delete(knowledge_base_id: str | list[str]) → None

Delete one or more knowledge bases.

list(project_id: str) → list[KnowledgeBase]

List knowledge bases, filtered by project.

list_documents(knowledge_base_id: str, topic_id: str | NotGiven = NOT_GIVEN) → list[Document]

List documents for a knowledge base, optionally filtered by topic.

retrieve(knowledge_base_id: str) → KnowledgeBase

Retrieve a knowledge base by ID.

update(knowledge_base_id: str, *, name: str | NotGiven = NOT_GIVEN, description: str | NotGiven = NOT_GIVEN, project_id: str | NotGiven = NOT_GIVEN) → KnowledgeBase

Update a knowledge base.

class giskard_hub.resources.ModelsResource(client: SyncClient)

Bases: APIResource

chat(model_id: str, messages: List[ChatMessage]) → ModelOutput

create(*, name: str, description: str, url: str, supported_languages: List[str], headers: Dict[str, str] = None, project_id: str) → Model

delete(model_id: str | List[str]) → None

list(project_id: str) → List[Model]

retrieve(model_id: str) → Model

update(model_id: str, *, name: str = NOT_GIVEN, description: str = NOT_GIVEN, url: str = NOT_GIVEN, supported_languages: List[str] = NOT_GIVEN, headers: Dict[str, str] = NOT_GIVEN, project_id: str = NOT_GIVEN) → Model

class giskard_hub.resources.ProjectsResource(client: SyncClient)

Bases: APIResource

create(*, name: str, description: str = '')

delete(project_id: str | List[str])

list()

retrieve(project_id: str)

update(project_id: str, *, name: str = NOT_GIVEN, description: str = NOT_GIVEN)

class giskard_hub.resources.ScheduledEvaluationsResource(client: SyncClient)

Bases: APIResource

Resource for managing scheduled evaluations.

create(*, project_id: str, name: str, model_id: str, dataset_id: str, frequency: Literal['daily', 'weekly', 'monthly'] | FrequencyOption, time: str, tags: List[str] | None = None, run_count: int = 1, day_of_week: int | None = None, day_of_month: int | None = None) → ScheduledEvaluation

Create a new scheduled evaluation.

Parameters:

• project_id (str) – The ID of the project to create the scheduled evaluation in.

• name (str) – The name of the scheduled evaluation.

• model_id (str) – The ID of the model to evaluate.

• dataset_id (str) – The ID of the dataset to evaluate against.

• frequency (str) – The frequency of the scheduled evaluation (daily, weekly, monthly).

• time (str) – The time to run the evaluation (HH:MM format).

• tags (List[str], optional) – List of tags to filter the conversations that will be evaluated.

• run_count (int, optional) – The number of times to run each test case (1-5), by default 1.

• day_of_week (int, optional) – The day of the week to run (1-7, 1 is Monday). Required for weekly frequency.

• day_of_month (int, optional) – The day of the month to run (1-28). Required for monthly frequency.

Returns:

The created scheduled evaluation.

Return type:

ScheduledEvaluation

delete(scheduled_evaluation_ids: str | List[str]) → None

Delete scheduled evaluations.

Parameters:

scheduled_evaluation_ids (Union[str, List[str]]) – List of scheduled evaluation IDs or a single scheduled evaluation ID to delete.

list(*, project_id: str) → List[ScheduledEvaluation]

List scheduled evaluations for a project.

Parameters:

project_id (str) – The ID of the project to list scheduled evaluations for.

Returns:

List of scheduled evaluations.

Return type:

List[ScheduledEvaluation]

list_evaluations(scheduled_evaluation_id: str) → List[EvaluationRun]

List evaluations linked to a scheduled evaluation.

Parameters:

scheduled_evaluation_id (str) – The ID of the scheduled evaluation to get evaluations for.

Returns:

List of evaluations linked to the scheduled evaluation.

Return type:

List[EvaluationRun]

retrieve(scheduled_evaluation_id: str) → ScheduledEvaluation

Retrieve a scheduled evaluation by ID.

Parameters:

scheduled_evaluation_id (str) – The ID of the scheduled evaluation to retrieve.

Returns:

The scheduled evaluation.

Return type:

ScheduledEvaluation

update(scheduled_evaluation_id: str, *, name: str | NotGiven = NOT_GIVEN, run_count: int | NotGiven = NOT_GIVEN, frequency: Literal['daily', 'weekly', 'monthly'] | FrequencyOption | NotGiven = NOT_GIVEN, time: str | NotGiven = NOT_GIVEN, day_of_week: int | NotGiven = NOT_GIVEN, day_of_month: int | NotGiven = NOT_GIVEN, paused: bool | NotGiven = NOT_GIVEN) → ScheduledEvaluation

Update a scheduled evaluation.

Parameters:

• scheduled_evaluation_id (str) – The ID of the scheduled evaluation to update.

• name (str, optional) – The name of the scheduled evaluation.

• frequency (str, optional) – The frequency of the scheduled evaluation (daily, weekly, monthly).

• time (str, optional) – The time to run the evaluation (HH:MM format).

• run_count (int, optional) – The number of times to run each test case (1-5).

• day_of_week (int, optional) – The day of the week to run (1-7, 1 is Monday). Required for weekly frequency.

• day_of_month (int, optional) – The day of the month to run (1-28). Required for monthly frequency.

• paused (bool, optional) – Whether the scheduled evaluation is paused.

Returns:

The updated scheduled evaluation.

Return type:

ScheduledEvaluation