Prompt Engineering and API Integration · Lesson 7
Understanding and Evaluating Generative AI Outputs
LLMs can be fluent and wrong. Hallucinations are confident statements not supported by facts or by the provided context. Outputs can also carry bias, leak sensitive data, or ignore format instructions.
Evaluate with a test set of real inputs and expected qualities. Methods include exact or fuzzy match for structured answers, metrics like BLEU/ROUGE for text overlap, human review, and LLM-as-a-judge where a strong model grades answers against a rubric.
Run evaluations every time you change a prompt, model or parameter, so you catch regressions before users do.
Key points
- Hallucination = confident but unsupported content
- Build a small, realistic evaluation set early
- LLM-as-judge scales review but needs a clear rubric
- Re-run evals after every prompt or model change