analiza aiqbench
Amazon Connect Talent applies industrial-organizational psychology principles to scale structured behavioral interviews using AI-driven adaptive questioning and evidence-based scoring. The system leverages job-specific competency frameworks to generate consistent evaluations that predict performance while reducing the manual overhead of training interviewers.
Inside Amazon Connect Talent: The science of structured AI interviews

De la AWS
Text integralFor organizations hiring hundreds or thousands of people, every interview presents the same challenge: How do you give candidates a consistent opportunity to demonstrate their skills while gathering the evidence needed to make informed hiring decisions?
Structured behavioral interviews offer a research-backed approach to this challenge. They use consistent, job-related questions and clear scoring criteria to give candidates an opportunity to demonstrate their skills. Decades of research in industrial-organizational (I-O) psychology have shown that structured interviews help organizations evaluate candidates and predict job performance (Schmidt & Hunter, 1998; Sackett et al., 2022). Yet delivering structured interviews at scale requires significant time and expertise to develop competencies, create questions, train interviewers, and evaluate responses consistently.
Amazon Connect Talent brings established hiring science to AI-led interviews at scale. Connect Talent uses structured questions, adaptive follow-ups, and evidence-based scoring to help organizations gather consistent, job-related information while giving more candidates the opportunity to demonstrate their strengths.
We built Connect Talent around three core design principles and evaluated how well the service applies them through a combination of internal research and independent expert review. The findings offer a closer look at the science behind Connect Talent’s AI-led, structured interviewing.
1. Build interviews around job-related evidence
Connect Talent builds each interview around the competencies required for the role. The service uses job requirements and established occupational frameworks to create behaviorally anchored scorecards. Each scorecard defines specific strength and concern signals, and every interview question targets the behaviors it aims to uncover.
The AI interviewer also adapts to each response. After every answer, it checks whether the candidate has provided enough evidence for the competency being assessed. When details are missing, it asks targeted follow-up questions. This approach preserves the structure of the interview while giving candidates more opportunities to explain their experience.
2. Separate interviewing from scoring
Connect Talent separates the interview conversation from the evaluation. The AI interviewer gathers evidence but does not assign scores. Afterward, a separate evaluation process identifies which behaviors the candidate demonstrated and cites the supporting transcript passa
3. Build fairness protections into the design
Connect Talent evaluates transcribed candidate responses without using video, voice tone, or physical appearance to calculate competency scores. The service also excludes candidate identity information from scoring and evaluates each competency separately. These safeguards help focus assessments on job-related behaviors and limit the influence of unrelated information.
To understand how well these design principles work in practice, we combined independent I-O psychology evaluations with internal research. We examined whether the service generates job-relevant interview content, identifies behavioral evidence in line with expert assessments, gathers more complete responses through follow-up questions, and distinguishes between levels of demonstrated competency.
The quality of a structured interview starts with the skills it measures. Candidates need questions that reflect the requirements of the role, and hiring teams need confidence that the interview assesses those skills.
Amazon engaged an independent I-O psychology firm to evaluate Connect Talent’s AI-generated interview content. The firm reviewed the service’s baseline output before human editing and examined how well its competencies, questions, and rubrics aligned with job requirements. The evaluation found that Connect Talent retained critical job information in 100% of the job descriptions reviewed. Of the competencies generated, 91% were clearly traceable to the job description. The firm’s psychologists also found that 98% of interview questions targeted the intended competencies, and 94% of rubric signals clearly measured the intended competency.
Relevant questions provide the foundation for a structured interview. The next challenge is evaluating the evidence candidates provide with consistency.
To examine this, an independent I-O psychology firm reviewed 36 interviews assessing whether candidates demonstrated specific strengths and concerns, creating 252 rubric signals.
Connect Talent evaluated the same transcripts, reviewed hundreds of rubric signals, and aligned with expert scoring decisions in 89.3% of cases. Scoring alignment reached 97.6% when candidates provided clear evidence and 92.9% when only considering candidate concern signals, which identify behaviors that may indicate a poor fit for the intended role.
The evaluation also produced a A Cohen’s kappa of 0.79 is just under the standard of 0.81 for almost perfect agreement. This outcome indicates substantial agreement between Connect Talent and the independent expert candidate scoring.
Candidates may have relevant experience but leave important details out of their first response. In a traditional one-way video interview, a candidate may receive a fixed question and have only one opportunity to answer it. Hiring teams must then evaluate the information available, even when the response leaves many important questions unanswered.
Connect Talent uses adaptive follow-ups to gather more complete evidence. For example, a candidate might describe improving a slow system. The AI interviewer can ask what changed, which actions the candidate took, and how they measured the improvement. These questions help candidates explain their contributions and results in greater detail.
We studied the impact of follow-up questions by comparing assessments based on a candidate’s first response with assessments based on the full interview conversation. In a preliminary evaluation, scores based only on the first response were up to 1.14 points lower on a five-point scale than scores based on the full conversation. Assessments using the full conversation also came closer to expert evaluations.
Without follow-up questions, the scoring process identified fewer strength signals and more concern signals. In some cases, the candidate’s initial response simply lacked the information needed to demonstrate a relevant behavior. The full conversation captured more of what candidates demonstrated and gave hiring teams a more complete record of their experience.
Hiring teams need scores that reflect meaningful differences in demonstrated skills. Yet research has found that some AI scoring approaches concentrate ratings within a narrow range, making those differences harder to recognize (Zhu et al., 2026).
Connect Talent uses behaviorally anchored scorecards to evaluate specific strength and concern signals. Rather than asking an AI model to assign an overall rating, the service identifies evidence for each behavior and calculates the competency score using a predefined formula.
Consider the competency “collaboration”. A general AI scoring approach might evaluate a candidate’s overall collaboration skills based on the interview transcript. Connect Talent examines specific behaviors, such as whether the candidate sought input from colleagues before making a decision. Each finding contributes to the final score.
We evaluated this approach across hundreds of expert-rated interview assessments. Connect Talent’s structured scoring used all five points of the rating scale, while holistic LLM-as-judge scoring concentrated its ratings within the middle three points.
This wider distribution allows the scoring system to represent more levels of demonstrated competency. The behaviorally anchored scorecards also help hiring teams understand the evidence behind each rating.
Connect Talent brings the rigor of structured interviewing to more people while giving each candidate a meaningful chance to demonstrate their skills. The result is a new opportunity for organizations hiring at scale: more candidates can show what they can do, and more hiring teams can make decisions grounded in the evidence that matters.
Join us at AWS re:Invent 2026 for BIZ332: Every Candidate, Every Time: AI-Led Structured Interviews with Amazon Connect Talent, an interactive Chalk Talk led by Sunitha Srivatsa and Gerald Friedland.
The session will explore the science behind Connect Talent, including structured interviewing, adaptive follow-up questions, and evidence-based scoring.
Session 1: Monday, November 30, 2026, 10:00–11:00 AM, Caesars Forum – Level 1, Academy 409.
Repeat session: Thursday, December 3, 2026, 3:00–4:00 PM, MGM Grand, Premier 309.
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research. Psychological Bulletin, 124(2), 262–274.
Booth, B. M., Hickman, L., Subburaj, S. K., Tay, L., Woo, S. E., & D’Mello, S. K. (2021). Bias and Fairness in Multimodal Machine Learning: A Case Study of Automated Video Interviews. Proceedings of the 2021 International Conference on Multimodal Interaction, 268–277. https://doi.org/10.1145/3462244.3479897
Zhu, K., Sheng, Z., Liu, Z., & Tan, J. (2026). Scale Collapse, Model Disagreement, and False Precision: A Multi-Model Audit of LLM Behavioral Interview Scoring.
Text integral sub licența deschisă a publicației · 1356 cuvinte
Subiecte conexe
When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews
arXiv · 18 sept. 2026
SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery
arXiv · 4 nov. 2026

LLM ensemble PREVIA predicts future PTSD risk from emergency department trauma narratives with expert-level accuracy
Nature · 9 nov. 2026
Accessible, but Not Adopted: Increasing LLM Adoption among First-generation, Low-income (FGLI) College Students beyond Expanding Access
arXiv · 30 sept. 2026