Quickstart & setup

Giskard Hub is our enterprise platform for LLM agent testing with team collaboration and continuous red teaming. The Hub provides a comprehensive user interface for performing LLM evaluations in production environments with enterprise-grade security and collaboration features.

The Hub is the user interface from which you can perform LLM evaluations. It implements the following 4-step workflow:

Create test datasets

Automate test case generation with a focus on legitimate and adversarial queries.

Create test datasets

Review tests with human feedback

Use domain knowledge to review and refine test cases through humans in the loop.

Review tests with human feedback

Run and schedule evaluations

Run evaluations and schedule them to run automatically.

Run and schedule evaluations

Compare evaluations

Compare evaluations to detect regressions and identify patterns.

Compare evaluation results

Continuous red teaming

Detect emerging vulnerabilities through proactive red teaming.

Continuous red teaming

Note

Throughout this user guide, we’ll use a banking app called Zephyr Bank, designed by data scientists. The app’s agent provides customer service support on their website, offering knowledge about the bank’s products, services, and more.

The dashboard

The Dashboard is the first page you’ll see upon logging in. It provides an overview of your project, displaying the number of agents, datasets, evaluations, and knowledge bases.

It also features a graph showing the agent’s performance over time, measured by the average success rate of the evaluations. The success rate is calculated based on some evaluation metrics, such as Conformity, Correctness, Groundedness, String Matching, Metadata, Semantic Similarity, and more. We’ll delve into these metrics in more detail in the Evaluations section.

Note

For detailed information about evaluation metrics and checks, including examples and how they work, see Review tests with human feedback.

Additionally, the dashboard lists your most recent evaluations and datasets for quick access.

Create a project

In this section, you will learn how to create a project.

First, click on the “Settings” icon on the left panel, this page allows you to manage your projects and users (if you have the proper access rights).

In the Projects tab, click on “Create project” button. A modal will appear where you can enter your project’s name and description.

Once the project is created, you can access its dashboard by clicking on it in the list. Alternatively, use the dropdown menu in the upper left corner of the screen to select the project you want to work on.

Setup an agent

This section guides you through creating a new agent.

Note

Agents are configured through an API endpoint. They can be evaluated against datasets.

On the Agents page, click on the “New agent” button.

The interface below displays the agent details that need to be filled out.

• Name: The name of the agent.

• Description: Used to refine automatic evaluation and generation for better accuracy in your specific use case.

• Supported Languages: Add the languages your agent can handle. Note that this affects data generation.

• Connection Settings:

  • Agent API Endpoint: The URL of your agent’s API endpoint. This is where requests are sent to interact with your agent.

  • Headers: These are useful for authentication and other custom headers

The endpoint should expect an object with the following structure:

{
    "messages": [
        {
        "role": "user",
        "content": "Hello!"
        },
        {
        "role": "assistant",
        "content": "Hello! How can I help you?"
        },
        {
        "role": "user",
        "content": "What color is an orange?"
        }
    ]
}

The endpoint’s response should have the following structure:

{
    "response": {
        "role": "assistant",
        "content": "An orange is green"
    },
    "metadata": {
        "some_key": "whatever value"
    }
}

Import a knowledge base

This section guides you through importing your custom knowledge base.

Note

A Knowledge Base is a domain-specific collection of information. You can have several knowledge bases for different areas of your business.

On the Knowledge Bases, click on “Add Knowledge Base” button.

The interface below displays the knowledge base details that need to be filled out.

• Name: The name of the knowledge base.

• File: The document to upload, containing the knowledge base content. Supported formats are:

  • JSON: A JSON file containing an array of objects

  • JSONL: A JSON Lines file with one object per line

  • CSV: A CSV File (Deprecated)

JSON/JSONL format requirements:

Each object in your JSON or JSONL file should have the following structure:

{
    "text": "Your document content here",
    "topic": "Optional topic classification"
}

• text (required): The document content

• topic (optional): The topic classification for the document

CSV format requirements:

Warning

Importing a knowledge base with CSV is deprecated and will be removed in future versions. Please use JSON or JSONL formats for new knowledge bases.

For CSV files, the format should have one column named text with the document content. If you’re uploading a knowledge base with pre-defined topics, the file should have two columns with the first row labeled text,topic.

General rules for all formats:

• If the text has a value but the topic is blank, the topic will be set to ‘Others’. However, if all topics are blank, the topic will be automatically generated.

• If both the text and topic are blank, or if the text is blank but the topic has a value, the entry will not be imported.

The interface below displays information about the knowledge base and its content with corresponding topics. As mentioned above, if no topics were uploaded with the knowledge base, Giskard Hub will also identify and generate them for you. In the example below, the knowledge base is ready to be used with over 1200 documents and 7 topics.