Machine Learning Interview Questions for Deployment

A machine learning interview should reveal how a candidate decides whether a model is ready for real users. Ask what information the model can use, what the evaluation actually measures and how the team would recover from a bad release. The fictional scenarios below give engineering managers ten questions with reference answers, follow-up probes and observable anchors.

TL;DR: Choose a role-relevant subset covering leakage, evaluation, model choice, deployment and monitoring. Listen for a causal explanation, a concrete next check and a safe decision; treat these editorial anchors as discussion guidance, not validated predictors of job performance.

---

Run a machine learning interview around decisions

Ashby's July 2026 report The Rise of AI in Job Postings examined over 200,000 qualifying postings from 2021 through Q2 2026 and found frequent AI references in engineering and data descriptions. Those are references in Ashby's dataset, not evidence that every engineering role needs ML expertise. Start from the decisions your hire will own.

• Set the context. Tell candidates the prediction time, intended users and cost of a mistake. Let them ask for missing information.
• Keep comparisons fair. Pick the same core scenarios and neutral probes for comparable candidates. Use additional questions to clarify evidence, not to rescue a preferred answer.
• Separate evidence from ratings. Write down what the candidate proposed and why. Mark an untested area as missing evidence rather than assuming competence or failure.

Use one shared weak anchor throughout: the candidate names a technique but cannot explain the failure mechanism or next check. Each question adds adequate and strong signals. These are illustrative anchors for your worksheet; adapt them to your role before the first interview.

Tip: Keep coding-language checks separate. Our Python interview questions cover language skills; the data engineer question bank covers pipeline decisions. This bank asks whether a model should make a live decision.

---

Machine learning interview questions on evaluation

The first questions test whether the reported result answers the deployment question. The scikit-learn guide to common pitfalls and recommended practices explains leakage and why learned preprocessing belongs inside the training boundary. Use the scenarios to ask candidates to locate that boundary themselves.

1. A churn model uses account status recorded after cancellation. What is wrong?

Reference answer: Ask when the prediction is made and which values existed then. A later cancellation status leaks the outcome. Rebuild features from information available at the decision time, remove the leaked status and rerun evaluation before interpreting the score.

• Probe: What if the column has an earlier event date but arrived in the system later?
• Adequate evidence: Identifies the unavailable feature and rebuilds the evaluation.
• Strong evidence: Distinguishes event time from availability time and checks other retrospective joins.

2. The same customers appear in training and test data. Is the evaluation valid?

Reference answer: It depends on the intended prediction. For unseen customers, hold customers apart. For future decisions about known customers, use a time-respecting split and only past information. Fit preprocessing within training folds; reserve an untouched final test. Check duplicates and label maturity.

• Probe: How would you evaluate both new customers and returning customers?
• Adequate evidence: Chooses the split from the deployment population and prediction time.
• Strong evidence: Designs separate population checks and prevents preprocessing and label leakage.

3. A rare-event classifier has high accuracy but misses most positive cases. What would you inspect?

Reference answer: Inspect the confusion matrix, positive-class recall, precision and the cost of each mistake. Evaluate candidate thresholds on validation data against review capacity and tolerated harm. Keep the final test separate from threshold selection; report the operating point and denominators rather than accuracy alone.

• Probe: What happens to your proposed threshold when the review queue is full?
• Adequate evidence: Explains missed positives and chooses metrics tied to the decision.
• Strong evidence: Balances review capacity, false alarms and missed cases without tuning on the final test.

Google's classification metrics guide explains why accuracy can conceal failure on rare positive cases. For the last scenario, ask the candidate to describe actual false positives and false negatives before naming a preferred metric.

Key takeaway: A strong evaluation answer identifies the prediction population, available information and decision cost. “Use cross-validation” alone leaves those questions unanswered.

---

Baselines and complex models: justify the extra burden

Google's Rules of ML recommends starting with simple approaches and meaningful metrics. The following fictional choices turn that advice into an interview discussion about the evidence needed for additional complexity.

4. What should a baseline establish before you try a complex model?

Reference answer: Use a simple rule or model that reflects the actual decision, with the same eligible inputs and evaluation split. Compare error costs, latency and operational effort. A majority-class predictor is a useful sanity check for classification, but may be useless for the positive cases the business needs.

• Probe: What would make you stop model development and keep the rule?
• Adequate evidence: Defines a comparable baseline and a useful decision metric.
• Strong evidence: Names a concrete improvement needed to justify added operational burden.

5. A complex model wins the average offline score. Would you ship it?

Reference answer: First ask about uncertainty, important customer groups, representative data and the size of the gain. Compare serving cost and latency with the baseline. Propose a bounded online comparison with a primary outcome and harm guardrails; an offline win alone does not establish useful customer impact.

• Probe: What if the gain disappears for your largest customer group?
• Adequate evidence: Checks more than the overall score before proposing deployment.
• Strong evidence: Connects a subgroup failure and an uncertain gain to a specific hold or rollout decision.

Tip: Give the candidate permission to keep the baseline. Ask what result would change that decision; do not reward complexity simply because it sounds advanced.

---

Deployment failure, rollout and rollback

Ask for a recovery path before asking for a launch date. Google's SRE Workbook chapter Canarying Releases describes a partial, time-limited deployment evaluated against a control. These scenarios extend that idea into model-specific questions about compatible inputs and decisions already made.

6. Offline evaluation passes, but live predictions disagree. Where do you start?

Reference answer: Compare a captured, permitted input through training and serving transformations. Check feature order, units, missing-value handling, preprocessing version and availability. Confirm that the expected model was loaded. Narrow exposure or use a safe fallback while locating the mismatch; do not immediately retrain on unexplained behavior.

• Probe: Which intermediate value would you compare first?
• Adequate evidence: Traces an input through both paths instead of guessing a new algorithm.
• Strong evidence: Isolates a concrete mismatch while limiting customer impact and protecting captured data.

7. How would you release a new model without exposing everyone at once?

Reference answer: Define a limited canary: a small, time-limited release compared with the current version. Agree what to measure, who watches and what stops expansion. Shadow predictions can check behavior without taking action, but cannot alone prove the effect of changing customer decisions.

• Probe: What can a shadow comparison tell you, and what can it not tell you?
• Adequate evidence: Limits initial exposure and defines a stop condition.
• Strong evidence: Separates observational shadow evidence from customer-impact evidence and plans staged expansion.

8. The new model causes harmful decisions. What must rollback restore?

Reference answer: Stop expansion and activate a rehearsed safe fallback or previous compatible bundle. Restore the model together with its expected preprocessing and decision threshold; check compatibility with current features. Assign an incident owner, verify recovery with live guardrails and address decisions already made. A version switch cannot undo earlier harm.

• Probe: What if the old model requires a feature that no longer exists?
• Adequate evidence: Restores a compatible decision path and checks recovery.
• Strong evidence: Handles an unusable previous version, names a fallback owner and considers prior affected decisions.

Key takeaway: “Roll back” is incomplete until the candidate explains what remains compatible, what fallback is safe and how the team verifies recovery.

---

Drift and monitoring after launch

Google's Monitoring pipelines guidance covers input validation, training-serving skew and quality checks across relevant data slices. Here, ask the candidate to separate an alert from evidence of a model problem. The monitoring plan below is an editorial exercise, not a built-in Intervy workflow.

9. What would you monitor when outcome labels arrive late?

Reference answer: Separate service health, data quality and eventual model quality. Track latency, failures, feature freshness, missingness and prediction distributions promptly. Join mature labels to the original predictions later and inspect quality by relevant group. Treat early proxies as warning signals, not proof of accuracy.

• Probe: How would you prevent partially observed outcomes from looking like better performance?
• Adequate evidence: Separates immediate health signals from delayed outcome evaluation.
• Strong evidence: Uses mature cohorts and investigates missing or selectively collected labels.

10. An input distribution changes. Should monitoring retrain the model automatically?

Reference answer: Investigate first: a distribution shift may reflect seasonality, a broken input or a changed population. It does not by itself establish that the relationship between inputs and outcomes changed. Check data integrity and labeled performance where available. Retrain only against an agreed evaluation and release process, retaining a fallback.

• Probe: What would you do if drift is large but labeled quality remains stable?
• Adequate evidence: Does not equate an input alert with proven model degradation.
• Strong evidence: Distinguishes data repair, continued observation and retraining with evidence for each action.

Tip: Ask who receives each alert and what action they can take. A dashboard with no response owner leaves an important part of the candidate's proposal unexplained.

---

Prepare the next interview and import the bank

For a fictional interview worksheet, select one evaluation question, one model-choice question, one deployment question and one monitoring question. Reserve time for probes and candidate questions. Before using the worksheet, have interviewers compare notes on a practice answer: which exact words support each anchor, and which evidence is still absent? See our structured technical interview guide for planning the broader conversation.

• Prepare the prompts. A technical interview preparation tool can hold your agreed question wording and reference answers. Keep the answer open to defensible alternatives, rather than demanding the paragraph above verbatim.
• Review the observations. When using a structured interview tool, keep your panel's discussion tied to recorded reasoning. These anchors do not automatically establish a hiring recommendation.

You can load all 10 questions from the Import section: choose "Machine Learning Deployment Judgment" in Sample Interview Data and select Load. Importing requires organization membership and permission to publish questions. Editing questions and their Example Answer requires the corresponding question-editing permission.

Key takeaway: Agree what evidence matters before interviewing. Use the bank to make the discussion concrete, then make the hiring decision against your role criteria.