Glossary
Glossary
Section titled “Glossary”This glossary provides formal definitions for all terms used throughout the IOA-ORM specification. Terms defined here are linked automatically when they appear in other sections.
IOA-ORM (Interactive Oral Assessment Ontology and Reference Model) : The primary artifact. A multi-layered specification for interactive oral assessment systems, comprising a domain ontology (shared vocabulary), a reference model (system abstraction), and an executable specification (machine-readable schema). See also: IOA Domain Ontology, IOA Reference Model, IOA Executable Specification.
IOA Domain Ontology : The conceptual layer of IOA-ORM. Defines the shared vocabulary, domain entities, and assessment-theoretic constructs (e.g., AssessmentProfile, EvidenceTarget, CompletionPolicy). Grounded in Joughin (1998), Akimov & Malin (2020), Bloom (1956), and Fenton (2025).
IOA Reference Model : The system abstraction layer of IOA-ORM. Defines the architecture (node graph, policies, evidence ledger), design principles, and runtime semantics for interactive oral assessment systems.
IOA Executable Specification : The machine-readable layer of IOA-ORM. Comprises TypeScript schema, validation rules, and the compilation pipeline from authoring to runtime execution.
Intermediate Representation (IR) : The engineering role played by the IOA Executable Specification — as a compilation target from authoring tools and a compilation source for runtime engines. “IR” describes a role, not the artifact’s primary identity.
IR : Shorthand for Intermediate Representation. See: Intermediate Representation.
Assessment Profile : A structured declaration of the exam’s position on Joughin’s (1998) six dimensions of oral assessment. Captures design parameters that determine what the exam measures and how validity/reliability claims are supported.
Assessment Purpose : Whether the exam is formative (practice/feedback), summative (graded), or diagnostic (placement). Affects evidence handling, feedback timing, and recording policy.
Assessment Validity : The degree to which an assessment measures what it claims to measure. The IOA-ORM addresses face validity, content validity, construct validity, concurrent validity, and fairness validity through structured evidence targets and Bloom’s taxonomy alignment.
Assessment-Significant Moments : Candidate behaviors detected during the exam that carry assessment meaning beyond content correctness — hesitation, self-correction, thinking-aloud. Emitted as distinct RuntimeEvent types (Fenton, 2025).
Autonomy Gradient : The spectrum describing the LLM examiner’s degree of control over different exam capabilities, ranging from fully autonomous (wording, dialogue strategy) through advisory (evidence sufficiency, anxiety detection) to fully controlled (transitions, timing) and forbidden (scoring, structural changes).
Best Effort
: A node completion status indicating the node ended with incomplete evidence (follow-ups exhausted or time budget hit). Partial evidence is still written to the Evidence Ledger with a completionStatus field.
Bloom’s Taxonomy : A classification of cognitive levels (Remember, Understand, Apply, Analyse, Evaluate, Create) used in the IOA-ORM to encode the cognitive depth of Evidence Targets and validate that exams test the intended cognitive level (Bloom, 1956).
Calibration Profile : References to calibration exercises and measured accuracy metrics for the AI examiner. Ensures consistent assessment quality across sessions.
Candidate Command : A structured input from the candidate that the runtime controller MUST process (e.g., repeat, pause, clarification). These are runtime primitives, not UI decorations.
Command Classifier : The runtime component that classifies candidate utterances into structured Candidate Commands (repeat, clarification, raise_hand) or normal answer responses. Operates on transcript text with confidence scores.
Completion Policy : Rules governing when a node is “done” — how many turns, what evidence is required, time limits. Machine-enforceable by the runtime controller.
Content Type : Joughin’s (1998) four primary categories of what oral assessment can measure: knowledge/understanding, applied problem solving, interpersonal competence, intrapersonal qualities.
Context Policy : Rules governing what exam context (rubric, previous nodes, candidate history) the AI examiner may access at each node.
Conversation Fingerprint : A SHA-256 hash of the ordered conversation path (node sequence + follow-up types + turn count per node) proving each exam instance is unique. Critical for academic integrity auditing.
Data Channel : The real-time communication bus (typically LiveKit data channel) between the exam bot and frontend. Used for low-latency event delivery of candidate commands, transcript deltas, and UI events.
Dialogue Move : A structural conversation act by the LLM examiner that precedes or bridges follow-ups, such as paraphrasing the candidate’s answer for confirmation. Does not count toward maxFollowUps.
Event Envelope : The universal container for all events emitted by any component. Carries eventId, sessionId, seq, timestamp, source, type, payload, and schemaVersion.
Event Store : The append-only persistence layer for Runtime Events. Provides audit trail, recovery capability, and session replay. All events are stored regardless of transport mechanism.
Evidence Coverage : The proportion of declared evidence targets that have been assessed (covered or not_covered) during a session. Used as a metric for node completion and assessment quality.
Evidence Dimension : The dimension of oral assessment an evidence signal addresses. Based on Joughin (1998) content types plus a fifth ‘metacognitive’ dimension (Fenton, 2025). Values: knowledge_understanding, applied_problem_solving, interpersonal_competence, intrapersonal_quality, metacognitive.
Evidence Gap
: A record that a mandatory evidence target has fewer positive signals than its minPositiveSignals threshold when a node exits. First-class data in the evidence ledger, not merely absence of signals.
Evidence Ledger : The authoritative, structured collection of all evidence signals produced during a session. First-class output consumed by the marking runtime.
Evidence Signal : A runtime-emitted record that a specific evidence target was (or was not) demonstrated, with confidence and provenance.
Evidence Target : A rubric-aligned definition of what the exam is trying to assess at a given node.
Evidence Vocabulary : Rubric criteria shared with the LLM examiner as a description of what to listen for during the assessment. Distinct from scoring logic — the LLM sees criteria (‘explains the mechanism’) but not how those criteria map to marks.
Exam : A published, versioned oral assessment with defined structure, policies, and evidence targets.
Exam Runtime Package
: The canonical machine-readable artifact representing a complete published exam. The single source of truth for all runtime configurations. Type name: ExamRuntimePackage. Also known as: Assessment Package, Published Package (once published and immutable).
Fairness Audit : Structured analysis of assessment outcomes across demographic dimensions to detect systematic disparities.
FlowManager : Pipecat Flows’ core component responsible for node management, context strategy, function registration, and pipeline orchestration. Driven by the Runtime Controller.
Follow-Up Policy : Rules governing how many follow-ups the examiner may issue, and under what conditions.
Guardrail : A runtime-enforced constraint that blocks or modifies LLM output before it reaches the candidate. Guardrails prevent rubric disclosure, score leakage, answer suggestion, and unauthorized transitions. Distinct from prompt-level instructions — guardrails are structural enforcement mechanisms that the LLM cannot override.
I (continued)
Section titled “I (continued)”Inter-Rater Reliability : The degree of agreement between independent markers assessing the same candidate performance. In the IOA-ORM, measured between AI-generated evidence signals and human-annotated signals, targeting Cohen’s κ ≥ 0.75.
Leading Question
: A question phrasing that suggests the correct answer (e.g., ‘Wouldn’t you say…’). Detected and blocked by the output validation pipeline’s leading_question filter.
Marking Package : The structured input assembled after exam completion, containing the evidence ledger, full transcript, runtime audit, and specification snapshot. Consumed by the Marking Runtime.
Marking Pipeline : The downstream pipeline that consumes the evidence ledger and produces assessment marks. Broader than the Marking Runtime alone — includes evidence ledger retrieval, signal aggregation, and score computation.
Marking Runtime : The downstream system that reads the evidence ledger and produces assessment scores.
Moderation Policy : Rules for human review of AI-generated evidence signals. Supports inter-rater reliability.
Node Kind : The type of a runtime node, serving as a discriminant for node behavior. Eight kinds: question, scenario, task, discussion, warmup, wrapup, branch, identity_check.
Output Validation Pipeline : A defense-in-depth pipeline that intercepts the LLM’s proposed spokenText before TTS. Four filters: content (forbidden phrases), topic (current node scope), action (no transitions/scoring), and length (max characters). Any failure triggers re-prompting.
Persona : A role the AI examiner adopts for scenario-based nodes (e.g., ‘hotel manager’). The Runtime validates that all LLM output stays in character. Persona breaks are intercepted and re-prompted.
Pipecat Adapter Output : The compiled Pipecat-specific configuration (FlowManager config + NodeConfig) generated from the IOA-ORM specification.
Prompting Level : A classification of examiner follow-up moves based on Pearce & Chiavaroli’s (2020) taxonomy: from neutral presentation to leading guidance.
Published Package : An immutable assessment package that has passed publish-time validation. Once published, its content MUST NOT change. New versions require new packageIds. See also: Exam Runtime Package.
Question Pool : A set of equivalent question variants from which one or more are drawn per session. Enables inter-case reliability.
Rapport Move : A dialogue move by the AI examiner that builds affective connection (encouragement, acknowledgement, reassurance) without providing evaluative feedback. Distinct from follow-ups; does not count toward maxFollowUps.
Recovery Policy : Rules governing how the runtime handles anomalies — silence, unclear answers, off-topic, anxiety, network issues.
Report Observation : The single function exposed to the LLM agent. Bundles all observations (evidence signals, command detection, answer quality, follow-up intent, spoken text) into one atomic call to the Runtime Controller.
Rubric Criteria : Observable competencies defined in the marking rubric that describe what candidates should demonstrate. Shared with the LLM as evidence vocabulary (what to listen for), distinct from scoring logic (how criteria map to marks).
Runtime Controller : The deterministic enforcement layer that sits between the LLM and the environment. Evaluates policies, approves transitions, validates evidence signals, enforces guardrails, and manages session state. The LLM proposes; the Runtime Controller disposes.
Runtime Event : An immutable record of a significant state change during a session.
Runtime Node : A discrete unit of the exam flow — a question, task, scenario segment, or transition point.
Runtime Session : A single candidate’s attempt at an exam. One exam may have many sessions.
Runtime State : The mutable, per-session state tracked by the runtime controller during execution.
Scaffolding : The examiner’s provision of graduated support to candidates within their Zone of Proximal Development. Operates at two levels: pre-exam familiarization (practice sessions) and in-assessment (adjusting support based on candidate performance). The amount of scaffolding provided is itself evidence of candidate competence (Vygotsky, 1978; Fenton, 2025).
Scaffolding Budget : Maximum scaffolding intensity permitted at a node (0–10). The amount of scaffolding provided is itself evidence of candidate competence (Fenton, 2025).
Scaffolding Intensity : A 0–3 scale measuring the degree of examiner support provided during a follow-up: 0 = no scaffolding (independent answer), 1 = minimal (rephrasing/redirect), 2 = moderate (conceptual hint), 3 = heavy (simplified question/broken into parts). Recorded on evidence signals as competence evidence.
Signal Kind : Classification of an evidence signal’s assessment value. Eight kinds: positive, partial, absent, misconception, flawed_reasoning, process_positive, process_negative, self_correction.
Telemetry Policy : Rules governing what operational data is emitted and where.
Three-Layer Authority : The architectural principle that authority is distributed across three exclusive layers: the Specification Layer (what the exam IS), the Runtime Controller Layer (what happens WHEN), and the LLM Agent Layer (what is SAID and HOW). Each layer has exclusive authority in its domain.
Transcript Turn : A single utterance in the conversation, attributed to examiner or candidate, with timing and node context.
Transition Policy : Rules governing how and when the runtime moves from one node to another.
Transversal Target : An evidence target that is assessed session-wide across all nodes rather than being scoped to specific nodes. Examples: communication quality, critical thinking. Uses aggregation methods: holistic, best_of, or trajectory.
Validity Claim : A structured declaration of how the exam addresses face, content, construct, concurrent, inter-rater, inter-case, or fairness validity.