Here's an uncomfortable truth: most technical interviews are barely better than flipping a coin. Research consistently shows that traditional coding interviews have weak correlation with on-the-job performance[^1].
After conducting thousands of engineering interviews and tracking hire outcomes, I've identified which questions actually predict success—and which are theater.
The Problem with Traditional Technical Interviews
Standard coding interviews test a narrow set of skills:
- Algorithm memorization
- Performance under artificial time pressure
- Whiteboard communication (a skill used nowhere else in engineering)
What they don't test:
- Ability to work with existing codebases
- Collaboration and communication
- Debugging and problem-solving in realistic conditions
- System design thinking
- Incremental delivery and iteration
Google's own internal research found that their famous brainteaser questions had "zero relationship" with job performance[^2]. They've since abandoned them—yet many companies still cargo-cult outdated practices.
What Actually Predicts Engineering Success
Research from decades of personnel psychology points to better predictors[^3]:
| Method | Validity Coefficient |
|---|---|
| Work sample tests | 0.54 |
| Structured interviews | 0.51 |
| Cognitive ability tests | 0.51 |
| Unstructured interviews | 0.38 |
| Job experience (years) | 0.18 |
Work samples—assessments that mirror actual job tasks—outperform everything else. Let's translate this into practical interview strategies.
Questions That Work: By Category
1. Past Behavior Questions (Highest Signal)
Past behavior is the best predictor of future behavior. Use the STAR format (Situation, Task, Action, Result) to probe deeply:
Technical Decision Making: "Tell me about a technical decision you made that you later realized was wrong. What was the context, what did you decide, what happened, and what did you learn?"
Why it works: Great engineers make mistakes but learn from them. This reveals self-awareness, growth mindset, and technical judgment.
Red flags: Candidates who can't name a mistake or who blame others entirely.
Debugging Under Pressure: "Describe the most difficult production bug you've debugged. Walk me through your investigation process from first alert to resolution."
Why it works: Reveals systematic thinking, tool familiarity, and how they perform when stakes are high.
Red flags: Vague descriptions, hero narratives without team acknowledgment, or inability to explain their methodology.
Technical Influence: "Tell me about a time you convinced your team to adopt a different technical approach. What was the situation, and how did you make your case?"
Why it works: Senior engineers need to influence without authority. This tests communication, persuasion, and technical credibility.
Red flags: Adversarial framing ("I had to fight the team"), or inability to recall an example.
2. System Design Questions (Essential for Senior Roles)
System design interviews, done well, reveal architectural thinking and tradeoff analysis. The key is making them collaborative rather than adversarial.
Effective framing: "Let's design a system together. I'll describe a problem we're actually facing, and I'd like to think through the solution with you. Feel free to ask clarifying questions—this is meant to be a conversation."
Good system design questions:
- Design a URL shortener (classic, well-calibrated difficulty)
- Design a real-time collaborative document editor
- Design a notification system that can send millions of messages daily
- Design the backend for a ride-sharing app
What to evaluate:
- Do they ask clarifying questions before diving in?
- Can they identify the core challenges and constraints?
- Do they consider scalability, reliability, and maintainability?
- Can they articulate tradeoffs between different approaches?
- Do they know what they don't know?
3. Code Review Questions (Underutilized)
Reviewing code is something senior engineers do daily. Yet few companies include it in interviews.
Format: Present a pull request with intentional issues—some obvious, some subtle. Ask the candidate to review it as they would for a colleague.
What to include:
- A subtle bug (off-by-one error, race condition)
- A performance issue
- A maintainability concern
- A security vulnerability
- Code style issues (to see if they bike-shed or focus on substance)
What to evaluate:
- Do they catch the important issues?
- How do they communicate feedback? (Constructive vs. harsh)
- Do they ask about context before making judgments?
- Can they distinguish critical issues from nitpicks?
4. Pair Programming (Best Work Sample)
Working on a problem together reveals more than any solo coding test.
Effective setup:
- Use a real problem from your codebase (simplified)
- Let them use their own environment and tools
- Play the role of helpful colleague, not silent evaluator
- Time-box appropriately (60-90 minutes max)
What to evaluate:
- How do they approach an unfamiliar codebase?
- Do they ask good questions?
- Can they collaborate and communicate their thinking?
- How do they handle getting stuck?
- Do they write code you'd want to maintain?
Questions to Stop Asking
Brainteasers
"How many golf balls fit in a school bus?"
These test nothing relevant and annoy candidates. Google abandoned them years ago.
Trivia
"What's the time complexity of quicksort's worst case?"
This tests memorization, not engineering ability. Anyone can look this up.
Gotcha Questions
"What's the output of this intentionally confusing JavaScript snippet?"
Language trivia doesn't predict job performance and signals that your culture values tricks over substance.
Puzzle Algorithms
Unless the role specifically requires algorithm design (and most don't), stop asking candidates to implement red-black trees or solve dynamic programming problems they'll never encounter on the job.
Structuring the Interview for Reliability
Interview reliability—whether different interviewers reach the same conclusion—matters as much as validity. Increase reliability by:
Use Scorecards
Define what "good" looks like before the interview. Score each competency independently (1-4 scale) with specific criteria.
Example scorecard for system design:
- Requirements gathering (1-4): Did they ask clarifying questions?
- Architecture (1-4): Did they propose a reasonable high-level design?
- Tradeoff analysis (1-4): Did they articulate pros/cons of different approaches?
- Depth (1-4): Could they dive deep on specific components when asked?
Independent Evaluation
Interviewers should submit their scores before discussing the candidate. Group discussions bias toward the first or loudest opinion.
Calibration Sessions
Regularly review past interview decisions against actual hire outcomes. Which questions predicted success? Which interviewers are too lenient or harsh?
Building Your Question Bank
Create a repository of validated questions that interviewers can draw from. For each question, document:
- The question text
- What competency it assesses
- What a strong answer looks like
- What a weak answer looks like
- Common follow-up questions
This ensures consistency across interviewers and makes it easy to train new interviewers.
Want help redesigning your technical interview process? Contact SmithSpektrum for a comprehensive interview audit.
References
[^1]: Maurer, S.D., "The Validity and Utility of Selection Methods in Personnel Psychology," Personnel Psychology, 2006 [^2]: Bryant, A., "In Head-Hunting, Big Data May Not Be Such a Big Deal," New York Times, June 2013 [^3]: Schmidt, F.L. & Hunter, J.E., "The Validity and Utility of Selection Methods," Psychological Bulletin, 1998