How do you approach prompt design for a new LLM-powered feature, from first draft to production?

I start by writing down the task specification before I write a single line of prompt: the exact input format, the exact output schema, and three or four worked examples of what a correct answer looks like, including at least one edge case. I draft the first version as a direct instruction with a clear role and constraints, then run it against a small hand-built test set of maybe twenty cases covering the range of real inputs, not just the easy ones. From there I iterate with a tight loop: change one variable at a time, whether that is instruction phrasing, example selection, or output format, and check it against the test set before moving to the next change, because stacking multiple edits at once makes it impossible to tell what actually helped. Once the prompt is stable on the hand-built set, I widen the test set using production-like data and run it through an eval framework such as promptfoo or Braintrust to catch regressions automatically. Only after that do I add structured output constraints and error handling for malformed responses. I keep every version in source control alongside the eval results, so a prompt change is reviewable the same way a code change is.

How do you evaluate whether a prompt change is actually an improvement?

A prompt change is only real if it moves a metric I defined before I made the change, so the first step is always having a fixed eval set and a scoring method in place. For tasks with a clear right answer, like extraction or classification, I score against exact match or a structured comparison. For open-ended generation, I use an LLM-as-judge setup with a detailed rubric, cross-checked against a human-labelled sample of at least thirty cases to confirm the judge model agrees with human judgement closely enough to trust it. I always run the new prompt and the old prompt against the identical eval set in the same run, not sequentially days apart, because model behaviour drifts even on a fixed prompt as providers update models silently behind an API. I look at the full distribution of scores, not just the average, since a change that helps the median case but breaks a specific category of edge cases is not a net improvement even if the mean goes up. Only once the eval set shows a clear, reproducible gain do I ship the change, and I keep the old prompt version tagged so I can roll back fast if production signals disagree with the offline eval.

How do you decide which model to use for a given task, and how do you handle model version changes?

Model choice comes down to task complexity, latency budget, and cost per request, roughly in that order. For simple classification or extraction tasks I default to a smaller, faster model, since a top-tier reasoning model is usually wasted spend for a task that does not need multi-step reasoning. For tasks that need genuine reasoning, tool use, or long-context synthesis, I reach for a frontier model and accept the latency and cost trade-off because getting the answer wrong costs more than the extra tokens. I benchmark two or three candidate models against my eval set before committing, rather than picking based on general reputation, because the model that performs best on public benchmarks is not always the one that performs best on my specific task and data. When a provider ships a new model version, I never let it auto-upgrade in production without running the full eval suite first, since even a minor version bump can shift behaviour enough to break a prompt that depended on the old model's specific quirks. I pin model versions explicitly in config and treat an upgrade as a deliberate, tested release, the same way I would treat a dependency bump in application code.

How do you reduce hallucination in an LLM's output?

I treat hallucination reduction as a layered problem rather than something a single prompt tweak fixes. At the prompt level, I instruct the model explicitly to say when it does not know something rather than guessing, and I give it permission to answer 'not enough information' as a valid output, since models often hallucinate because the prompt implicitly demands an answer. For anything factual, I ground the response in retrieved source material rather than relying on the model's parametric knowledge, and I instruct it to cite the specific passage it used, which both improves accuracy and gives me a way to verify the answer automatically. I use structured output formats wherever possible, because forcing the model into a schema reduces the surface area for confident-sounding fabrication. For high-stakes outputs I add a self-verification pass, a second call that checks the first response against the source material and flags claims that are not supported. None of these are perfect individually, but layered together they catch most of what a single well-written prompt misses, and I track the residual hallucination rate through my eval set so I know when the layered approach is actually working rather than just adding latency.

Tell me about a time a prompt that worked well in testing failed in production.

I built a prompt that classified inbound support tickets into one of twelve categories, and it scored above 95% accuracy against my test set of two hundred labelled examples. In production, accuracy dropped to around 78% within the first week. When I dug into the failures, the pattern was clear: my test set had been built from historical tickets that were all in English, but roughly a quarter of live traffic came in French and Spanish, and the model was defaulting to a generic 'other' category whenever the input language shifted, something the original prompt never addressed because I had not thought to check the language distribution of real traffic against my test set. I rebuilt the eval set to match the real language mix, added explicit multilingual examples to the prompt, and added a language detection step upstream so the classification prompt received a language hint rather than guessing. Accuracy recovered to 93% across all three languages within a few days. The lesson was that a test set built from a convenient sample, rather than one that actually reflects production distribution, will always overstate how good a prompt is.

Describe a situation where you had to push back on a product request that would have made a prompt unreliable or unsafe.

A product manager wanted the support assistant to give definitive answers on whether a specific action would affect a customer's warranty, phrased with full confidence rather than hedged language, because user testing showed hedged answers frustrated people. The problem was that warranty terms varied by product line and purchase date, and our retrieval system did not always have complete coverage of edge-case policies, so a confidently wrong answer carried real financial and trust risk. I put together a short analysis showing the categories of warranty questions where our retrieval confidence was actually high versus the categories where it was thin, and proposed a middle path: confident, direct answers for the high-confidence categories, and an explicit 'let me connect you with a specialist' handoff for the thin-coverage categories, rather than a blanket policy of confident answers everywhere. I backed this with the actual retrieval confidence scores from a two-week sample rather than an abstract risk argument, which made the trade-off concrete for the product manager instead of theoretical. The product team accepted the tiered approach, and it shipped without a single warranty-related complaint in the following quarter.

Give an example of how you improved an underperforming prompt or AI feature.

I inherited a prompt for generating meeting summaries that users rated 2.9 out of 5 on average, with the most common complaint being that summaries missed action items buried in casual conversation rather than stated formally. I pulled fifty of the lowest-rated transcripts and read them manually rather than guessing at the fix, and found that the original prompt asked the model to extract 'action items' without defining what counted as one, so it consistently missed anything phrased as a suggestion or a question rather than a direct instruction. I rewrote the prompt with an explicit definition of an action item, including examples of the indirect phrasing it needed to catch, and split the task into two passes: first extract every candidate commitment or task-like statement, then a second pass to filter and format the confirmed action items. I validated the fix against the fifty transcripts, then against the full eval set, before rolling it to production behind a feature flag for a week. The user rating rose to 4.3 out of 5, and the specific complaint about missed action items dropped to near zero in the following month's feedback.

Walk me through how you would design an eval suite for a customer-facing AI feature before it ships.

I start by defining the failure modes that actually matter for the specific feature, not a generic quality score, because 'good' means something different for a summarisation feature than for a data extraction feature. I build the test set from three sources: hand-written cases covering known edge cases, a sample of real or realistic production-like inputs, and adversarial cases designed to break the prompt, including attempts at prompt injection if the feature takes user text as input. I define scoring per failure mode rather than one aggregate number: factual accuracy scored against source material, format compliance checked programmatically since that does not need a judge model, tone and helpfulness scored with LLM-as-judge against a rubric, and safety checked against a fixed set of known problematic inputs. I calibrate the LLM-as-judge scores against a human-labelled sample before trusting them in the pipeline, typically thirty to fifty cases scored by both, and I do not proceed until agreement is high enough to trust the automated score as a proxy. I wire this into CI so every prompt change runs the full eval automatically, and I set a hard gate: no prompt change ships if it regresses any individual failure mode, even if the aggregate score improves.

How do you manage prompt versioning and rollback in production?

I treat prompts as versioned artefacts with the same discipline as application code, stored in source control rather than a database field that gets edited in place, so every change has a diff, an author, and a linked eval result. In production I keep prompts behind a config layer that can reference a specific version by ID, which lets me run a percentage rollout or an A/B test between two prompt versions without a code deploy. Every prompt version is tagged with the eval results it passed and the model version it was validated against, so if a model provider ships an update, I know immediately which prompt versions need re-validation rather than discovering it from a spike in error reports. For rollback, I keep the previous stable version live and reachable through the same config layer, so reverting is a config change rather than a deploy, which gets a bad prompt out of production in minutes rather than waiting for a release cycle. I also log the prompt version ID alongside every production request, so when a user reports a bad output, I can reproduce it against the exact prompt and model version that generated it rather than the current version, which might already differ.

How do you approach prompting for an agentic workflow that needs to call tools and make multi-step decisions?

I start by defining the tool schemas as precisely as I would an API contract, with tight parameter types and clear descriptions, because vague tool descriptions are the single biggest cause of an agent calling the wrong tool or passing malformed arguments. I keep the system prompt focused on the agent's decision policy, when to call a tool versus answer directly, when to ask the user for clarification versus proceeding on an assumption, and what to do when a tool call fails, rather than trying to enumerate every possible scenario. I build in explicit stopping conditions and a maximum step count, because an under-specified agent will happily loop through unproductive tool calls if nothing tells it to stop and reassess. For multi-step tasks I favour an explicit plan-then-execute pattern over pure reactive tool calling, where the model first states its intended sequence of steps before executing them, since this makes the reasoning inspectable and catches obviously wrong plans before any tool actually runs. I log the full tool call trace for every agent run, not just the final answer, because when an agent fails, the useful debugging signal is almost always in the sequence of intermediate decisions, not the final output alone. I evaluate agent runs on task completion rate and on the number of unnecessary tool calls, since an agent that succeeds by brute-forcing ten redundant calls is not actually production-ready.

Prompt Engineer

Prompt Engineer interviews test something narrower and harder than most people expect: whether you can get consistent, measurable behaviour out of a model that is probabilistic by nature, and prove it with an eval rather than a feeling. Interviewers push on how you version prompts, how you catch a regression before it reaches users, and how you reason about the trade-off between a bigger frontier model and a smaller, cheaper one for a specific task. Expect real 2026-era ground: agentic tool use, structured outputs, hallucination mitigation, and what you do when a provider quietly updates a model under your prompt. This guide covers the questions that come up most often, with answers that read like someone who ships prompts to production, not someone who has read about it.

For general interview preparation tips, read our guide to common interview questions.

Prepare further

How to ace the behavioural interview (STAR method)How to prepare for a technical interview

Common Prompt Engineer Interview Questions

Behavioural Interview Questions for Prompt Engineer Roles

Technical Questions for Prompt Engineer Candidates

What Hiring Managers Look for in Prompt Engineer Interviews

What hiring managers really look for in Prompt Engineer candidates:

Evidence over intuition. Candidates who can describe an eval set and a scoring method for every prompt claim are far more credible than candidates who say a prompt 'just works better'.
Awareness that models drift under you. Providers update models silently, and candidates who pin versions and re-validate on upgrade have clearly been burned by this before, which is exactly the experience you want.
Real production debugging habits. Prompt versioning, logging, and rollback discipline are what separate someone who can maintain a live AI feature from someone who can only demo one.
Judgement about model selection, not default use of the biggest model available. The strongest candidates treat model choice as a cost and latency decision, not a reflex.
Fluency with current tooling and techniques, not a 2023-era mental model of what a prompt is. Ask about agentic workflows and structured outputs specifically. If the answer is vague, the candidate's experience may not be current.

Questions to Ask Your Interviewer

→What does the current eval and testing setup look like for prompts here, and how mature is it?
→How do you handle it when a model provider ships a new version that changes behaviour under an existing prompt?
→How much of this role is pure prompt design versus the surrounding system: retrieval, tool integration, evaluation infrastructure?
→What is the biggest prompt failure the team has shipped to production, and what changed as a result?
→How are decisions made about which model to use for a given feature, and who owns that call?

Practise These Questions Before Your Interview

The mock interview tool builds a practice session around a specific job posting and your background, so you rehearse the questions most likely to come up.

Start Practising

Free on your first tracked role.

Related Roles

AI Engineer

Tech10 questions

Machine Learning Engineer

Available in Other Languages

Français Español