​
Set up Maihem

A

Generate API key

Generate API key in your Maihem account settings.

Book a call to get access to the Maihem platform

B

Set API key as environment variable

Set your Maihem API key as an environment variable called MAIHEM_API_KEY.

C

Install or upgrade the Python SDK

This assumes you have Python installed. See this guide to install Python if needed.

pip install --upgrade maihem

Make sure to continuously upgrade to the latest version

​
Integrate with codebase

1

Add target agent

In the terminal run the following command to add a description of your target agent. This will also create a folder in the current directory.

Command line interface (CLI)
maihem target-agent add \
--name "demo-financial-assistant" \
--role "AI Financial Assistant" \
--description "An AI assistant that provides information and summaries from financial documents."

Go to the new folder demo-financial-assistant, where there are some pre-populated Python scripts.

Command line interface (CLI)
cd demo-financial-assistant
2

Add decorators to each step of the workflow of your target agent

File demo-agent-workflow.py contains a pre-built workflow for the demo target agent. A decorator is added to each step of the workflow to map and evaluate each step.

demo-agent-workflow.py
import maihem
from maihem.evaluators import ContextRetrievalEvaluator, AnswerGenerationEvaluator
import requests

class Agent:

	@maihem.workflow_step(target_agent_name="demo-financial-assistant")
	def run_workflow(conversation_id: str, message: str) -> str:
		"""Trigger workflow to generate a response"""

		@maihem.workflow_step(evaluator=ContextRetrievalEvaluator())
		def context_retrieval(conversation_id: str, maihem_agent_message: str) -> str:
			"""Retrieve a list of chunks to be used as context for the LLM."""
			payload = {"conversation_id": conversation_id, "message": maihem_agent_message}
			url = "https://demo-agent.maihem.ai/context-retrieval"
			response = requests.request("POST", url, json=payload).json()
			return response["contexts"]

		@maihem.workflow_step(evaluator=AnswerGenerationEvaluator())
		def generate_answer(conversation_id: str, maihem_agent_message: str, contexts: List) -> str:
			"""Generate a response using a list of retrieved contexts"""
			payload = {"conversation_id": conversation_id, "message": maihem_agent_message, "contexts": contexts}
			url = "https://demo-agent.maihem.ai/reply"
			response = requests.request("POST", url, json=payload).json()
			return response["message"]

		contexts = context_retrieval(conversation_id, message)
		message = generate_answer(conversation_id, message, contexts)
		return message

agent = Agent()

​
Test RAG workflow

3

Create test with auto-generated data

Generate a dynamic test set of simulated conversations that will be used to evaluate your target agent. The test set is dynamic so it will adapt to different responses from your target agent.

Command line interface (CLI)
maihem test create auto-generated-data \
--name "test-1" \
--target-agent-name "demo-financial-assistant" \
--maihem-behavior-prompt "Ask questions about historical returns of stock in the S&P 500 during the last 10 years" \
--number-conversations 4
4

Run test

Command line interface (CLI)
maihem test run \
--name "modelX_prompt2_5_28-11-2024" \
--test-name="test-1"
5

See results in Maihem UI

Go to your Maihem account to see the results of your first test.

​
Next step: see our how-to guides

Test for hallucinations with simulated data

Evaluate workflow with your data

Monitor RAG workflow in production

Get Maihem SDK and API keyWith auto-generated dataset