What is Maihem?

Maihem creates AI agents that automatically test your LLM applications, by simulating conversations between your Target Agent and Maihem Agents.

The benefits of Maihem are:

Develop 10x times faster by automatically finding failures

Build LLM applications that you and others can trust

Get started

1

Get access

Book a call to get access to the Maihem platform.

2

Generate API key

Sign into your Maihem account and generate an API key by going to your account settings.

3

Set API key as environment variable

Before using the maihem package, you need to set your maihem API key as an environment variable:

Environment variable: MAIHEM_API_KEY

4

Install the Python SDK

5

Choose Dev Mode or Test Mode

Dev Mode

Simulate a single conversation to quickly debug your Target Agent, by highly customizing how the Maihem Agent behaves.

Test Mode

Simulate many conversations in parallel to generate a test set, and evaluate the performance of your Target Agent across different metrics.

6

Create config file

Create a config.yaml file, where you will pass the parameters that will define the simulations.

  • In Dev Mode, you need to define at least one Target Agent and one Maihem Agent.

  • In Test Mode, you need to define at least one Target Agent and one Test.

7

Create Maihem script

Before you run the script, you need to set the following:

  • Select parameters
    • Dev Mode or Test Mode
    • target_agent_identifier
    • (For Dev Mode) maihem_agent_identifier
    • (For Test Mode) test_identifier
    • Path to config file config_path
  • Implement chat function (wrap you target agent in the function provided in the script)

Copy the script below and save it as run_maihem.py:

Change the default values in the script or pass them

8

Run Maihem script

9

See results in Maihem webapp

Go to your account to visualize the conversations and evaluations.