Agents & Knowledge Bases

Agents

An Agent is your LLM application. The Hub calls your agent’s HTTP endpoint during evaluations and scans.

Register a remote agent

from giskard_hub import HubClient

hub = HubClient()

agent = hub.agents.create(
    project_id="project-id",
    name="Support Bot v2",
    description="GPT-4o chatbot with RAG over the product knowledge base",
    url="https://your-app.example.com/api/chat",
    supported_languages=["en", "fr"],
    headers=[{"name": "Authorization", "value": "Bearer <token>"}],
)

print(agent.id)

The Hub sends a POST request to url with a JSON body containing a messages array of {role, content} objects. Your endpoint must return a JSON object with a message field.

Test the connection

Before running an evaluation, verify your agent endpoint is reachable and responds correctly:

ping = hub.agents.test_connection(
    url="https://your-app.example.com/api/chat",
    headers={"Authorization": "Bearer <token>"},
)

print(ping.response)

Generate a completion

You can invoke a registered agent directly from the SDK without running a full evaluation:

output = hub.agents.generate_completion(
    "agent-id",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
    ],
)

print(output.response)
print(output.metadata)  # any metadata returned by your agent

Auto-generate a description

If your agent’s description is missing or stale, the Hub can generate one by observing how the agent behaves:

description = hub.agents.autofill_description("agent-id")
hub.agents.update("agent-id", description=description)

Using a local Python function as an agent

For evaluations where you don’t want to expose an HTTP endpoint — for example, when evaluating a model locally during development — pass a Python callable to hub.evaluations.create_local(). See Evaluations for details.

List, update, and delete agents

agents = hub.agents.list(project_id="project-id")

hub.agents.update("agent-id", name="Support Bot v2.1")

hub.agents.delete("agent-id")

---

Knowledge Bases

A Knowledge Base is an indexed collection of text documents. It has three primary uses in the Hub:

1. Document-based dataset generation — the Hub uses your documents as source material to auto-generate realistic test cases.
2. Grounded vulnerability scans — probes are anchored to your actual content, making attacks more realistic and specific to your domain.
3. Groundedness check context — retrieve relevant documents via hub.knowledge_bases.search_documents() and pass them as the context field of a groundedness check assertion to verify that your agent’s responses are grounded in your actual documents rather than hallucinated content.

Create a knowledge base

Documents are provided as a JSON or JSONL file where each record has a text field and an optional topic field.

From a Python list (in-memory)

import json

documents = [
    {"text": "Our return policy allows returns within 30 days of purchase.", "topic": "Returns"},
    {"text": "Free shipping is available on all orders over $50.", "topic": "Shipping"},
    {"text": "You can track your order via the link in your confirmation email.", "topic": "Shipping"},
]

kb = hub.knowledge_bases.create(
    project_id="project-id",
    name="Product Documentation",
    description="Official product docs and FAQs",
    file=("documents.json", json.dumps(documents).encode("utf-8")),
)

print(kb.id)

From a file on disk

from pathlib import Path

kb = hub.knowledge_bases.create(
    project_id="project-id",
    name="Product Documentation",
    description="Official product docs and FAQs",
    file=Path("documents.json"),
)

Note

After creation, the Hub indexes the documents asynchronously. Depending on the size of your corpus, this may take a few seconds to a few minutes. Check kb.status before using the KB for generation or scanning.

Retrieve and update a knowledge base

kb = hub.knowledge_bases.retrieve("kb-id")
print(kb.name, kb.status.state)

hub.knowledge_bases.update("kb-id", name="Updated Name")

Search documents

You can perform a semantic search over the documents in a knowledge base directly from the SDK:

results = hub.knowledge_bases.search_documents(
    "kb-id",
    search="return policy",
    limit=5,
)

for doc in results:
    print(doc.snippet)

Retrieve a specific document

doc = hub.knowledge_bases.retrieve_document("kb-id", "document-id")
print(doc.content)

List and delete knowledge bases

kbs = hub.knowledge_bases.list(project_id="project-id")

hub.knowledge_bases.delete("kb-id")

---

Using a knowledge base for dataset generation

Once your KB is ready, pass its ID to hub.datasets.generate_document_based() to create test cases grounded in your documents:

dataset = hub.datasets.generate_document_based(
    project_id="project-id",
    knowledge_base_id="kb-id",
    agent_id="agent-id",
    dataset_name="FAQ-based test suite",
    n_examples=20,
)

print(f"Generated dataset: {dataset.id} ({dataset.name})")

The Hub samples documents from the KB, crafts questions whose answers are grounded in those documents, and creates test cases with a groundedness check pre-configured.

See Datasets, Test Cases & Checks for more detail.

---

Using a knowledge base in a vulnerability scan

Pass a knowledge_base_id when creating a scan to run probes that are grounded in your documents. This makes adversarial attacks more domain-specific and increases detection accuracy for RAG-based systems:

scan = hub.scans.create(
    project_id="project-id",
    agent_id="agent-id",
    knowledge_base_id="kb-id",
    tags=["gsk:threat-type='hallucination'"], # Hallucination
)

See Vulnerability Scanning for the full list of tags and scan options.