Manage your projects, agents and knowledge bases

Projects are the top-level organizational units in Giskard Hub. They provide a workspace for your team to collaborate on LLM agent testing and evaluation.

Each project can contain:

  • Agents: The AI systems you want to test and evaluate

  • Datasets: Collections of test cases and conversations

  • Knowledge bases: Domain-specific information sources

  • Evaluations: Test runs and their results

  • Users and groups: Team members with different access levels

In this section, we will walk you through how to manage projects using the SDK.

Let’s start by initializing the Hub client or take a look at the Quickstart & setup section to see how to install the SDK and connect to the Hub.

from giskard_hub import HubClient

hub = HubClient()

You can now use the hub client to create, update, and delete projects, agents, and knowledge bases.

Projects

Create a project

You can create a project using the hub.projects.create() method. Example:

project = hub.projects.create(
    name="My first project",
    description="This is a test project to get started with the Giskard Hub client library",
)

Retrieve a project

You can retrieve a project using the hub.projects.retrieve() method:

project = hub.projects.retrieve("<PROJECT_ID>")

Update a project

You can update a project using the hub.projects.update() method:

project = hub.projects.update("<PROJECT_ID>", name="My updated project")

You can also update the project description and failure categories:

from giskard_hub.data.project import FailureCategory

# Define failure categories for your project
failure_categories = [
    FailureCategory(
        identifier="hallucination",
        title="Hallucination",
        description="Model generated false or misleading information"
    ),
    FailureCategory(
        identifier="bias",
        title="Bias",
        description="Model response shows unfair bias"
    )
]

# Update project with failure categories
project = hub.projects.update(
    "<PROJECT_ID>",
    name="My updated project",
    description="Updated project description",
    failure_categories=failure_categories
)

Delete a project

You can delete a project using the hub.projects.delete() method:

hub.projects.delete("<PROJECT_ID>")

List projects

You can list all projects using the hub.projects.list() method:

projects = hub.projects.list()
for project in projects:
    print(project.name)

Working with Failure Categories

Failure categories help you organize and classify different types of AI failures that occur during evaluations. They provide structured categorization of issues like hallucinations, bias, or other business logic failures.

from giskard_hub.data.project import FailureCategory

# Access failure categories from a project
project = hub.projects.retrieve("<PROJECT_ID>")

for category in project.failure_categories:
    print(f"Category: {category.title}")
    print(f"Identifier: {category.identifier}")
    print(f"Description: {category.description}")

When you run evaluations, failed test cases can be automatically or manually assigned to these failure categories, helping you track and analyze patterns in your AI system’s failures.

Agents

Create an agent

You can create an agent using the hub.models.create() method. Example:

model = hub.models.create(
    project_id=project.id,
    name="My Agent",
    description="An agent for demo purposes",
    url="https://my-agent-endpoint.example.com/agent_v1",
    supported_languages=["en", "fr"],
    # if your agent endpoint needs special headers:
    headers={"X-API-Key": "MY_TOKEN"},
)

After creating the agent, you can test it by running a chat:

response = model.chat(
    messages=[
        dict(role="user", content="What is the capital of France?"),
        dict(role="assistant", content="Paris"),
        dict(role="user", content="What is the capital of Germany?"),
    ],
)
print(response)

If all is working well, this will return something like:

ModelOutput(
    message=ChatMessage(
        role='assistant',
        content='The capital of Germany is Berlin.'
    ),
    metadata={}
)

Retrieve an agent

You can retrieve an agent using the hub.models.retrieve() method:

model = hub.models.retrieve("<MODEL_ID>")

Update an agent

You can update an agent using the hub.models.update() method:

model = hub.models.update("<MODEL_ID>", name="My updated agent")

Delete an agent

You can delete an agent using the hub.models.delete() method:

hub.models.delete("<MODEL_ID>")

List agents

You can list all agents in a project using the hub.models.list() method:

models = hub.models.list("<PROJECT_ID>")
for model in models:
    print(model.name)

Knowledge bases

The hub.knowledge_bases resource allows you to create, retrieve, update, delete, and list knowledge bases, as well as list topics and documents within a knowledge base.

Create a knowledge base

You can create a knowledge base using the hub.knowledge_bases.create() method. The data parameter can be a path (relative or absolute) to a JSON/JSONL file or a list of dicts containing a text key and an optional topic key.

# Create a kb from a file
kb_from_file = hub.knowledge_bases.create(
    project_id="<PROJECT_ID>",
    name="My knowledge base",
    data="my_kb.json",  # could also be a JSONL file
    description="A knowledge base for finance domain",
)

kb_from_list = hub.knowledge_bases.create(
    project_id="<PROJECT_ID>",
    name="My knowledge base",
    data=[
        {"text": "The capital of France is Paris", topic="europe"},
        {"text": "The capital of Germany is Berlin", topic="europe"}
    ],
    description="A knowledge base for geography domain",
)

After creating the knowledge base, we need to wait for it to be ready because we need to process documents and topics server-side:

kb.wait_for_completion()

Retrieve a knowledge base

You can retrieve a knowledge base by ID:

kb = hub.knowledge_bases.retrieve("<KNOWLEDGE_BASE_ID>")

Update a knowledge base

You can update a knowledge base:

kb_updated = hub.knowledge_bases.update(
    "<KNOWLEDGE_BASE_ID>",
    name="Updated KB name",
    description="Updated description"
)

Delete a knowledge base

You can delete a knowledge base by ID (or a list of IDs):

hub.knowledge_bases.delete("<KNOWLEDGE_BASE_ID>")

List knowledge bases

You can list all knowledge bases in a project:

kbs = hub.knowledge_bases.list(project_id=project.id)
for kb in kbs:
    print(f"{kb.name} - Topics: {[topic['name'] for topic in kb.topics]}")

List documents in a knowledge base

You can list documents for a knowledge base, optionally filtered by topic:

documents = hub.knowledge_bases.list_documents("<KNOWLEDGE_BASE_ID>")
for doc in documents:
    print(doc.content)

# To filter by topic:
documents = hub.knowledge_bases.list_documents("<KNOWLEDGE_BASE_ID>", topic_id="<TOPIC_ID>")
for doc in documents:
    print(doc.content)