Manage your projects, agents and knowledge bases

Projects are the top-level organizational units in Giskard Hub. They provide a workspace for your team to collaborate on LLM agent testing and evaluation.

Each project can contain:

  • Agents: The AI systems you want to test and evaluate

  • Datasets: Collections of test cases and conversations

  • Knowledge bases: Domain-specific information sources

  • Evaluations: Test runs and their results

  • Users and groups: Team members with different access levels

In this section, we will walk you through how to manage projects using the SDK.

Let’s start by initializing the Hub client or take a look at the Quickstart & setup section to see how to install the SDK and connect to the Hub.

from giskard_hub import HubClient

hub = HubClient()

You can now use the hub client to create, update, and delete projects, agents, and knowledge bases.

Projects

Create a project

You can create a project using the hub.projects.create() method. Example:

project = hub.projects.create(
    name="My first project",
    description="This is a test project to get started with the Giskard Hub client library",
)

Retrieve a project

You can retrieve a project using the hub.projects.retrieve() method:

project = hub.projects.retrieve("<PROJECT_ID>")

Update a project

You can update a project using the hub.projects.update() method:

project = hub.projects.update("<PROJECT_ID>", name="My updated project")

Delete a project

You can delete a project using the hub.projects.delete() method:

hub.projects.delete("<PROJECT_ID>")

List projects

You can list all projects using the hub.projects.list() method:

projects = hub.projects.list()
for project in projects:
    print(project.name)

Agents

Create an agent

You can create an agent using the hub.models.create() method. Example:

model = hub.models.create(
    project_id=project.id,
    name="My Agent",
    description="An agent for demo purposes",
    url="https://my-agent-endpoint.example.com/agent_v1",
    supported_languages=["en", "fr"],
    # if your agent endpoint needs special headers:
    headers={"X-API-Key": "MY_TOKEN"},
)

After creating the agent, you can test it by running a chat:

response = model.chat(
    messages=[
        dict(role="user", content="What is the capital of France?"),
        dict(role="assistant", content="Paris"),
        dict(role="user", content="What is the capital of Germany?"),
    ],
)
print(response)

If all is working well, this will return something like:

ModelOutput(
    message=ChatMessage(
        role='assistant',
        content='The capital of Germany is Berlin.'
    ),
    metadata={}
)

Retrieve an agent

You can retrieve an agent using the hub.models.retrieve() method:

model = hub.models.retrieve("<MODEL_ID>")

Update an agent

You can update an agent using the hub.models.update() method:

model = hub.models.update("<MODEL_ID>", name="My updated agent")

Delete an agent

You can delete an agent using the hub.models.delete() method:

hub.models.delete("<MODEL_ID>")

List agents

You can list all agents in a project using the hub.models.list() method:

models = hub.models.list("<PROJECT_ID>")
for model in models:
    print(model.name)

Knowledge bases

The hub.knowledge_bases resource allows you to create, retrieve, update, delete, and list knowledge bases, as well as list topics and documents within a knowledge base.

Create a knowledge base

You can create a knowledge base using the hub.knowledge_bases.create() method. The data parameter can be a path (relative or absolute) to a JSON/JSONL file or a list of dicts containing a text key and an optional topic key.

# Create a kb from a file
kb_from_file = hub.knowledge_bases.create(
    project_id="<PROJECT_ID>",
    name="My knowledge base",
    data="my_kb.json",  # could also be a JSONL file
    description="A knowledge base for finance domain",
)

kb_from_list = hub.knowledge_bases.create(
    project_id="<PROJECT_ID>",
    name="My knowledge base",
    data=[
        {"text": "The capital of France is Paris", topic="europe"},
        {"text": "The capital of Germany is Berlin", topic="europe"}
    ],
    description="A knowledge base for geography domain",
)

After creating the knowledge base, we need to wait for it to be ready because we need to process documents and topics server-side:

kb.wait_for_completion()

Retrieve a knowledge base

You can retrieve a knowledge base by ID:

kb = hub.knowledge_bases.retrieve("<KNOWLEDGE_BASE_ID>")

Update a knowledge base

You can update a knowledge base:

kb_updated = hub.knowledge_bases.update(
    "<KNOWLEDGE_BASE_ID>",
    name="Updated KB name",
    description="Updated description"
)

Delete a knowledge base

You can delete a knowledge base by ID (or a list of IDs):

hub.knowledge_bases.delete("<KNOWLEDGE_BASE_ID>")

List knowledge bases

You can list all knowledge bases in a project:

kbs = hub.knowledge_bases.list(project_id=project.id)
for kb in kbs:
    print(f"{kb.name} - Topics: {[topic['name'] for topic in kb.topics]}")

List documents in a knowledge base

You can list documents for a knowledge base, optionally filtered by topic:

documents = hub.knowledge_bases.list_documents("<KNOWLEDGE_BASE_ID>")
for doc in documents:
    print(doc.content)

# To filter by topic:
documents = hub.knowledge_bases.list_documents("<KNOWLEDGE_BASE_ID>", topic_id="<TOPIC_ID>")
for doc in documents:
    print(doc.content)