The research behind every metric.

As conversational AI systems are deployed in role-constrained settings (customer support agents, educational tutors, medical assistants), measuring whether a model consistently adheres to its assigned role becomes a core quality requirement. This paper proposes Role Adherence, a metric that quantifies role consistency across multi-turn conversations under a constructive definition: an assistant must actively exhibit the behaviour its role demands, not merely avoid out-of-scope content.

Evaluating the quality of a system prompt for a large language model (LLM) is a non-trivial task. Existing approaches often collapse quality into a single scalar score derived from a single LLM judge call, which lacks scientific grounding and diagnostic value.

Most agent evaluation frameworks score each conversation turn independently and report a single accuracy number. This hides two critical production concerns: how often a fully correct end-to-end conversation can be expected across repeated attempts, and where exactly tool usage breaks down when it does.