.svg){kind=logo}
Overview
Metric collection
Overview
Metric collection
Collection of available metrics to evaluate your LLM application with Maihem
These metrics are grouped into three primary categories - quality, safety, and security – which are then divided into further sub-categories:
- Quality Assurance (QA)
- Safety
- Security
Quality Assurance (QA)
Customer Experience (CX)
| Metric name | ID | Description |
|---|---|---|
| Goal Completion | qa_cx_goal_completion | If the Target Agent enabled the Maihem Agent to complete its goal. |
| Helpfulness | qa_cx_helpfulness | If the Target Agent’s response was helpful. |
| Retention | qa_cx_retention | If the Maihem Agent would like to use the Target Agent in the future. |
| Net Promoter Score (NPS) | qa_cx_nps | If the Maihem Agent would recommend the Target Agent. |
Retrieval-Augmented Generation (RAG)
| Metric name | ID | Description |
|---|---|---|
| Hallucination | qa_rag_hallucination | If a claim by the Target Agent is not supported or contradicted by the context. A claim refers to a statement in the output of the Target Agent. Context refers to the information that the Target Agent retrieved from a knowledge source (e.g. a vector database) to formulate its response. |
| Answer Relevance | qa_rag_answer_relevance | If the answer of the Target Agent is relevant to the previous question of the Maihem Agent. |
Safety
Bias
| Metric name | ID | Description |
|---|---|---|
| Disability | sec_bias_disability | If the Target Agent is biased against disabled people. |
| Gender | sec_bias_gender | If the Target Agent is biased against the female gender. |
| Physical body | sec_bias_physical_body | If the Target Agent is biased against different body types. |
| Politics | sec_bias_politics | If the Target Agent is biased against different political views. |
| Race | sec_bias_race | If the Target Agent is biased against different races. |
| Religion | sec_bias_religion | If the Target Agent is biased against different religions. |
Brand reputation
| Metric name | ID | Description |
|---|---|---|
| Competitor Recommendation | sec_brand_competitor_recommendation | If the Target Agent recommends competitors. |
| Negative Sentiment | sec_brand_negative_sentiment | If the Target Agent speaks negatively about its own brand. |
Toxicity
Coming soon!
Illegal content
Coming soon!
Security
Overreach
| Metric name | ID | Description |
|---|---|---|
| Financial advice | sec_overreach_financial_advice | |
| Legal advice | sec_overreach_legal_advice | |
| Medical advice | sec_overreach_medical_advice | |
| Unauthorized access | sec_overreach_unauthorized_access |
Privacy (PII)
| Metric name | ID |
|---|---|
| Address | sec_pii_address |
| sec_pii_email | |
| Name | sec_pii_name |
| Phone | sec_pii_phone |
System access
Coming soon!