A scorecard is useful when a reviewer can explain why a candidate received a rating. A total such as 78 out of 100 is much less useful when nobody can trace it back to a response, a criterion and a consistent standard.
Short answer: Build an AI interview scorecard around observable, job-related evidence. Define scoring anchors before the interview, distinguish missing evidence from a weak answer and retain a human review step for uncertain or consequential conclusions.
What we verified: The assessment principles here draw on OPM's structured-interview guidance. The sample support-role rubric is an original teaching example. Its scale and thresholds have not been validated for employee selection.
What belongs on a usable scorecard?
Include the competency, question, response evidence, rating anchor, uncertainty and reviewer decision. A score without supporting evidence should trigger a request for clarification rather than become a convenient default.
Record the rubric version and the relevant question version. If you change what counts as a strong answer midway through a hiring batch, preserve that change and decide whether earlier assessments need review. Otherwise, identical-looking scores can represent different standards.
For a broader introduction to the interview process, start with our AI interview guide.
How do you turn a job requirement into a criterion?
Start with a task a successful employee must perform. Suppose a support representative must investigate billing problems without promising an unauthorized refund. The requirement can become three observable criteria: identifying the issue, gathering relevant information and choosing a permitted next action.
Avoid measuring whatever happens to be easy for a model to describe. Professional tone might matter in a particular role, but an unexplained personality score or an inference about motivation is not a substitute for demonstrated behavior.
OPM describes structured interviews as measuring job-related competencies with predetermined questions and common rating standards. Use that principle to connect the question to the criterion, rather than generating a scorecard after reading the answers. Source: OPM.
What would a simple rubric look like?
Consider this hypothetical prompt: A customer says they were charged twice and wants a refund today. Explain what you would check and what you would tell them before taking action.
| Criterion | Needs more evidence | Demonstrated | Strong evidence |
|---|---|---|---|
| Clarifies the issue | Assumes every duplicate-looking entry is a duplicate charge | Checks the relevant transaction details | Also distinguishes pending and settled records where relevant |
| Handles uncertainty | Promises a result before checking | Explains the checks and avoids unsupported promises | Explains a clear next step if the first explanation is wrong |
| Respects authority | Issues an unauthorized commitment | Uses the relevant escalation or refund policy | Explains what they can do now and what needs approval |
The table intentionally avoids a hiring cutoff. A teaching rubric can demonstrate the structure without pretending that a score of two automatically means someone should advance.
How should the evidence be recorded?
Original explanatory diagram. The sample rubric is illustrative and requires role-specific review.
For each criterion, keep a short evidence excerpt or a traceable response reference. Then explain how that evidence matches the anchor. Do not write a positive-sounding summary that adds facts the candidate did not provide.
For example: “They said they would check transaction dates and consult the refund policy before making a promise” supports an authority-related rating. “They have excellent customer empathy and would be a great cultural fit” introduces a broader conclusion without the same evidential foundation.
Distinguish the original response from an AI-generated summary. A reviewer needs access to the source when the summary is disputed.
How do you handle missing or unclear answers?
Use a separate state for insufficient evidence. A broken microphone, ambiguous transcript or unanswered question is not necessarily equivalent to demonstrating poor performance.
A useful review record asks: Was the question delivered? Was the response captured? Did the answer address the criterion? Is a permitted clarification needed? The sequence prevents a capture failure from silently becoming a score.
Define the same clarification policy for everyone. A neutral probe such as “What information would you check first?” can establish missing reasoning. A leading probe such as “Wouldn't you check whether the charge is still pending?” may supply the answer you intended to assess.
This is also relevant to interview-integrity concerns. Uncertainty should lead to proportionate verification, not an unsupported cheating label.
How should AI assist with scoring?
Give the system the approved rubric and require evidence for each proposed rating. Tell it to mark missing evidence rather than infer experience, credentials or intent.
Review the supplied interview response against the approved rubric only.
For each criterion, return:
1. The candidate's relevant words or a response reference.
2. The matching rubric anchor.
3. A proposed rating, or insufficient evidence.
4. Any ambiguity requiring a human reviewer.
Do not infer protected characteristics, honesty, personality, qualifications or experience that the response does not establish. Do not make an advance or reject decision.
This is an editorial prompt template, not a validated scoring model. Test it with your own approved examples, including cases that should produce insufficient evidence.
How do you calibrate reviewers?
Have reviewers score the same example responses independently, then discuss disagreements. Focus on why an anchor was interpreted differently. If reviewers disagree about the rubric, adding an automated scorer may make that ambiguity harder to see.
Keep examples for boundary cases: a concise but sufficient answer, a polished but irrelevant answer and an answer that needs clarification. Record corrections and update the guide when the criterion genuinely requires clarification.
Do not choose thresholds simply because the distribution looks tidy. If you need a selection threshold, establish and evaluate it using appropriate assessment expertise and applicable hiring requirements. This article does not supply a universal pass mark.
What should the hiring manager receive?
Provide a compact report with the criteria, evidence, unresolved questions and reviewer conclusions. Show why a follow-up is recommended. Avoid giving the impression that a precise number measures a person's overall worth or predicts their performance with certainty.
For agency work, separate the candidate's evidence from the recruiter's recommendation and the client's decision. That makes it easier to discuss disagreements without rewriting the interview history. Our job-brief-to-shortlist workflow shows how this report fits into the larger process.
Your next steps
- Choose one role task and define the evidence it requires.
- Write a question and a small set of clear scoring anchors.
- Include insufficient evidence as an explicit outcome.
- Score examples independently before involving automation.
- Preserve evidence and reasons for reviewer overrides.
Download the scorecard design checklist.
Frequently asked questions
Should every competency have the same weight?
Not automatically. Weighting should reflect the role and the purpose of the assessment. Explain why one competency matters more and evaluate whether the weighting is appropriate. Equal weights are also a design choice, not a neutral default.
Can we score communication from an interview transcript?
You can assess evidence relevant to a defined communication task, but a transcript does not capture every aspect of communication. Do not infer unrelated traits from grammar, accent, word choice or transcription errors. Review what the role actually requires.
How many questions should an AI interview have?
Enough to obtain the evidence required for its purpose, with a reasonable burden on candidates. There is no universal number. Remove repeated questions that measure the same thing without resolving a genuine uncertainty.