Prompt Engineer

I start by writing down the task specification before I write a single line of prompt: the exact input format, the exact output schema, and three or four worked examples of what a correct answer looks like, including at least one edge case. I draft the first version as a direct instruction with a clear role and constraints, then run it against a small hand-built test set of maybe twenty cases covering the range of real inputs, not just the easy ones. From there I iterate with a tight loop: change one variable at a time, whether that is instruction phrasing, example selection, or output format, and check it against the test set before moving to the next change, because stacking multiple edits at once makes it impossible to tell what actually helped. Once the prompt is stable on the hand-built set, I widen the test set using production-like data and run it through an eval framework such as promptfoo or Braintrust to catch regressions automatically. Only after that do I add structured output constraints and error handling for malformed responses. I keep every version in source control alongside the eval results, so a prompt change is reviewable the same way a code change is.

A prompt change is only real if it moves a metric I defined before I made the change, so the first step is always having a fixed eval set and a scoring method in place. For tasks with a clear right answer, like extraction or classification, I score against exact match or a structured comparison. For open-ended generation, I use an LLM-as-judge setup with a detailed rubric, cross-checked against a human-labelled sample of at least thirty cases to confirm the judge model agrees with human judgement closely enough to trust it. I always run the new prompt and the old prompt against the identical eval set in the same run, not sequentially days apart, because model behaviour drifts even on a fixed prompt as providers update models silently behind an API. I look at the full distribution of scores, not just the average, since a change that helps the median case but breaks a specific category of edge cases is not a net improvement even if the mean goes up. Only once the eval set shows a clear, reproducible gain do I ship the change, and I keep the old prompt version tagged so I can roll back fast if production signals disagree with the offline eval.

Model choice comes down to task complexity, latency budget, and cost per request, roughly in that order. For simple classification or extraction tasks I default to a smaller, faster model, since a top-tier reasoning model is usually wasted spend for a task that does not need multi-step reasoning. For tasks that need genuine reasoning, tool use, or long-context synthesis, I reach for a frontier model and accept the latency and cost trade-off because getting the answer wrong costs more than the extra tokens. I benchmark two or three candidate models against my eval set before committing, rather than picking based on general reputation, because the model that performs best on public benchmarks is not always the one that performs best on my specific task and data. When a provider ships a new model version, I never let it auto-upgrade in production without running the full eval suite first, since even a minor version bump can shift behaviour enough to break a prompt that depended on the old model's specific quirks. I pin model versions explicitly in config and treat an upgrade as a deliberate, tested release, the same way I would treat a dependency bump in application code.

I treat hallucination reduction as a layered problem rather than something a single prompt tweak fixes. At the prompt level, I instruct the model explicitly to say when it does not know something rather than guessing, and I give it permission to answer 'not enough information' as a valid output, since models often hallucinate because the prompt implicitly demands an answer. For anything factual, I ground the response in retrieved source material rather than relying on the model's parametric knowledge, and I instruct it to cite the specific passage it used, which both improves accuracy and gives me a way to verify the answer automatically. I use structured output formats wherever possible, because forcing the model into a schema reduces the surface area for confident-sounding fabrication. For high-stakes outputs I add a self-verification pass, a second call that checks the first response against the source material and flags claims that are not supported. None of these are perfect individually, but layered together they catch most of what a single well-written prompt misses, and I track the residual hallucination rate through my eval set so I know when the layered approach is actually working rather than just adding latency.

I built a prompt that classified inbound support tickets into one of twelve categories, and it scored above 95% accuracy against my test set of two hundred labelled examples. In production, accuracy dropped to around 78% within the first week. When I dug into the failures, the pattern was clear: my test set had been built from historical tickets that were all in English, but roughly a quarter of live traffic came in French and Spanish, and the model was defaulting to a generic 'other' category whenever the input language shifted, something the original prompt never addressed because I had not thought to check the language distribution of real traffic against my test set. I rebuilt the eval set to match the real language mix, added explicit multilingual examples to the prompt, and added a language detection step upstream so the classification prompt received a language hint rather than guessing. Accuracy recovered to 93% across all three languages within a few days. The lesson was that a test set built from a convenient sample, rather than one that actually reflects production distribution, will always overstate how good a prompt is.

A product manager wanted the support assistant to give definitive answers on whether a specific action would affect a customer's warranty, phrased with full confidence rather than hedged language, because user testing showed hedged answers frustrated people. The problem was that warranty terms varied by product line and purchase date, and our retrieval system did not always have complete coverage of edge-case policies, so a confidently wrong answer carried real financial and trust risk. I put together a short analysis showing the categories of warranty questions where our retrieval confidence was actually high versus the categories where it was thin, and proposed a middle path: confident, direct answers for the high-confidence categories, and an explicit 'let me connect you with a specialist' handoff for the thin-coverage categories, rather than a blanket policy of confident answers everywhere. I backed this with the actual retrieval confidence scores from a two-week sample rather than an abstract risk argument, which made the trade-off concrete for the product manager instead of theoretical. The product team accepted the tiered approach, and it shipped without a single warranty-related complaint in the following quarter.

I inherited a prompt for generating meeting summaries that users rated 2.9 out of 5 on average, with the most common complaint being that summaries missed action items buried in casual conversation rather than stated formally. I pulled fifty of the lowest-rated transcripts and read them manually rather than guessing at the fix, and found that the original prompt asked the model to extract 'action items' without defining what counted as one, so it consistently missed anything phrased as a suggestion or a question rather than a direct instruction. I rewrote the prompt with an explicit definition of an action item, including examples of the indirect phrasing it needed to catch, and split the task into two passes: first extract every candidate commitment or task-like statement, then a second pass to filter and format the confirmed action items. I validated the fix against the fifty transcripts, then against the full eval set, before rolling it to production behind a feature flag for a week. The user rating rose to 4.3 out of 5, and the specific complaint about missed action items dropped to near zero in the following month's feedback.

I start by defining the failure modes that actually matter for the specific feature, not a generic quality score, because 'good' means something different for a summarisation feature than for a data extraction feature. I build the test set from three sources: hand-written cases covering known edge cases, a sample of real or realistic production-like inputs, and adversarial cases designed to break the prompt, including attempts at prompt injection if the feature takes user text as input. I define scoring per failure mode rather than one aggregate number: factual accuracy scored against source material, format compliance checked programmatically since that does not need a judge model, tone and helpfulness scored with LLM-as-judge against a rubric, and safety checked against a fixed set of known problematic inputs. I calibrate the LLM-as-judge scores against a human-labelled sample before trusting them in the pipeline, typically thirty to fifty cases scored by both, and I do not proceed until agreement is high enough to trust the automated score as a proxy. I wire this into CI so every prompt change runs the full eval automatically, and I set a hard gate: no prompt change ships if it regresses any individual failure mode, even if the aggregate score improves.

I treat prompts as versioned artefacts with the same discipline as application code, stored in source control rather than a database field that gets edited in place, so every change has a diff, an author, and a linked eval result. In production I keep prompts behind a config layer that can reference a specific version by ID, which lets me run a percentage rollout or an A/B test between two prompt versions without a code deploy. Every prompt version is tagged with the eval results it passed and the model version it was validated against, so if a model provider ships an update, I know immediately which prompt versions need re-validation rather than discovering it from a spike in error reports. For rollback, I keep the previous stable version live and reachable through the same config layer, so reverting is a config change rather than a deploy, which gets a bad prompt out of production in minutes rather than waiting for a release cycle. I also log the prompt version ID alongside every production request, so when a user reports a bad output, I can reproduce it against the exact prompt and model version that generated it rather than the current version, which might already differ.

I start by defining the tool schemas as precisely as I would an API contract, with tight parameter types and clear descriptions, because vague tool descriptions are the single biggest cause of an agent calling the wrong tool or passing malformed arguments. I keep the system prompt focused on the agent's decision policy, when to call a tool versus answer directly, when to ask the user for clarification versus proceeding on an assumption, and what to do when a tool call fails, rather than trying to enumerate every possible scenario. I build in explicit stopping conditions and a maximum step count, because an under-specified agent will happily loop through unproductive tool calls if nothing tells it to stop and reassess. For multi-step tasks I favour an explicit plan-then-execute pattern over pure reactive tool calling, where the model first states its intended sequence of steps before executing them, since this makes the reasoning inspectable and catches obviously wrong plans before any tool actually runs. I log the full tool call trace for every agent run, not just the final answer, because when an agent fails, the useful debugging signal is almost always in the sequence of intermediate decisions, not the final output alone. I evaluate agent runs on task completion rate and on the number of unnecessary tool calls, since an agent that succeeds by brute-forcing ten redundant calls is not actually production-ready.