Overview
Metric collection
Collection of available metrics to evaluate your LLM application with Maihem
These metrics are grouped into three primary categories - quality, safety, and security – which are then divided into further sub-categories:
- Quality Assurance (QA)
- Safety
- Security
Quality Assurance (QA)
Customer Experience (CX)
Metric name | ID | Description |
---|---|---|
Goal Completion | qa_cx_goal_completion | If the Target Agent enabled the Maihem Agent to complete its goal. |
Helpfulness | qa_cx_helpfulness | If the Target Agent’s response was helpful. |
Retention | qa_cx_retention | If the Maihem Agent would like to use the Target Agent in the future. |
Net Promoter Score (NPS) | qa_cx_nps | If the Maihem Agent would recommend the Target Agent. |
Retrieval-Augmented Generation (RAG)
Metric name | ID | Description |
---|---|---|
Hallucination | qa_rag_hallucination | If a claim by the Target Agent is not supported or contradicted by the context. A claim refers to a statement in the output of the Target Agent. Context refers to the information that the Target Agent retrieved from a knowledge source (e.g. a vector database) to formulate its response. |
Answer Relevance | qa_rag_answer_relevance | If the answer of the Target Agent is relevant to the previous question of the Maihem Agent. |
Safety
Bias
Metric name | ID | Description |
---|---|---|
Disability | sec_bias_disability | If the Target Agent is biased against disabled people. |
Gender | sec_bias_gender | If the Target Agent is biased against the female gender. |
Physical body | sec_bias_physical_body | If the Target Agent is biased against different body types. |
Politics | sec_bias_politics | If the Target Agent is biased against different political views. |
Race | sec_bias_race | If the Target Agent is biased against different races. |
Religion | sec_bias_religion | If the Target Agent is biased against different religions. |
Brand reputation
Metric name | ID | Description |
---|---|---|
Competitor Recommendation | sec_brand_competitor_recommendation | If the Target Agent recommends competitors. |
Negative Sentiment | sec_brand_negative_sentiment | If the Target Agent speaks negatively about its own brand. |
Toxicity
Coming soon!
Illegal content
Coming soon!
Security
Overreach
Metric name | ID | Description |
---|---|---|
Financial advice | sec_overreach_financial_advice | |
Legal advice | sec_overreach_legal_advice | |
Medical advice | sec_overreach_medical_advice | |
Unauthorized access | sec_overreach_unauthorized_access |
Privacy (PII)
Metric name | ID |
---|---|
Address | sec_pii_address |
sec_pii_email | |
Name | sec_pii_name |
Phone | sec_pii_phone |
System access
Coming soon!