How do you decide whether to use a pre-trained LLM directly, fine-tune one, or build a RAG system?

The decision comes down to data access patterns, latency requirements, and the nature of the knowledge the system needs. If the task is generative and the base model already has the domain knowledge baked in, prompt engineering with a hosted model like GPT-4o or Claude is usually the fastest path to value. Fine-tuning makes sense when you have high-quality labelled examples of the exact output format you want and need consistent stylistic control, or when latency and cost require a smaller, specialised model. RAG is the right choice when the system needs access to recent, proprietary, or frequently changing information that the base model cannot have seen in training. In practice, many production systems combine all three: a fine-tuned model for tone and format, retrieval for live knowledge, and careful prompt design to wire them together.

Walk me through how you would design a RAG pipeline for a customer support application.

I would start with the retrieval layer. For a support use case, the knowledge base typically includes product documentation, past resolved tickets, and policy documents. I would chunk the content carefully, aiming for 300 to 500 token chunks with meaningful overlap, and embed each chunk using a model like text-embedding-3-large from OpenAI or a hosted alternative. I would store embeddings in a vector database such as Pinecone or pgvector, and add metadata filters for product category and language to narrow retrieval before the semantic search runs. On the generation side, I would use a two-stage approach: first retrieve the top five chunks by cosine similarity, re-rank them with a cross-encoder, then pass the top three into the final generation prompt with clear instructions to cite the source chunk. I would also add a query classification step to route obvious FAQ questions directly without retrieval, reducing latency and cost for the majority of queries.

How do you evaluate the quality of an LLM-powered feature?

Evaluation for LLM systems has to happen at two levels. The first is offline evaluation against a fixed dataset of inputs and expected outputs, which I build before I build the feature. I create a test set of at least 50 representative cases covering typical inputs, edge cases, and known failure modes. I score outputs using a combination of automated metrics (BLEU or ROUGE for extractive tasks, LLM-as-judge scoring for open-ended generation) and human review of a random sample. The second level is online evaluation in production via user signals: thumb ratings, edit rates if users can modify AI output, escalation rates for support use cases, and session abandonment. I instrument these signals from day one because offline evals never perfectly predict user experience. I also set up a latency and cost dashboard, since a feature that degrades because a model API call starts timing out will not appear in quality metrics at all.

How do you handle prompt injection and adversarial inputs in production AI systems?

Prompt injection is one of the more serious risks in any system where user input is incorporated into a prompt. My standard approach is a layered defence. At the input layer, I validate and sanitise user inputs before they reach the prompt, flagging strings that contain instruction-like patterns such as "ignore previous instructions" or role-switching language. In the prompt itself, I use strict role framing and XML-like delimiters to separate trusted system content from untrusted user content. At the output layer, I apply output validation to catch responses that deviate from the expected schema or contain suspicious content before they reach the user. For higher-risk applications I also log all inputs and outputs for audit, and I run a periodic red-teaming exercise against the production prompt to surface new injection vectors. None of these defences are perfect in isolation, which is why the layered approach matters.

Tell me about a time an AI feature you built did not perform as expected in production.

I built a document summarisation feature for an internal knowledge management tool. In testing it performed well, producing accurate, concise summaries across the sample documents we used. After launch, users reported that summaries of long legal contracts were missing key clauses. The root cause was a chunking strategy that split documents at fixed token boundaries rather than at semantic boundaries like section headings, so the retrieval step was feeding the model partial clauses without context. I rewrote the chunker to respect document structure, using headings and numbered list patterns in the XML source to create logical chunks rather than fixed-size ones. Post-fix, the missing clause rate dropped from roughly 18% of documents to under 3%. The lesson was that chunking strategy is not a footnote in a RAG system: it is as important as the model choice.

Describe how you have handled a disagreement with a product team about what an AI system should and should not do.

The product team at my previous company wanted our AI assistant to answer questions about competitor pricing, using information scraped from public websites. I had concerns about accuracy because competitor pricing changes frequently, and about the reputational risk of the assistant confidently stating figures that were out of date. I put together a short risk assessment covering the accuracy half-life of the data, the support ticket volume we would likely see when users acted on stale prices, and the legal risk of quoting competitor figures in certain regulated markets. The product team accepted the accuracy argument but still wanted some competitor awareness in the product. We agreed on a compromise: the assistant would acknowledge questions about competitors and redirect to a human for detailed comparisons. It was a better outcome than either of the original positions.

Give an example of how you have made an AI system more cost-efficient without sacrificing quality.

We were running all queries through GPT-4o and our monthly API spend had grown to a point where the unit economics did not work at the pricing tier we were targeting. I ran a query classification analysis on a 2,000-query sample and found that around 60% of queries were simple factual lookups that did not need a large model. I introduced a routing layer that sent simple queries to GPT-4o mini and reserved the full model for complex multi-step reasoning tasks and edge cases flagged by a low confidence score. I validated the routing against a human-labelled test set before deploying. The result was a 54% reduction in API cost with quality scores on the evaluation set dropping by less than 1%. The routing logic became a template used across three other AI features in the product.

How do you structure an ML inference pipeline for low-latency production use?

The key is to separate the concerns of preprocessing, model inference, and postprocessing into distinct, independently scalable stages. For preprocessing I keep transformations stateless so they can run in parallel across request batches. For inference I use an optimised serving framework such as vLLM or TGI for LLMs, which handles continuous batching and KV-cache reuse to dramatically increase throughput compared to a naive HTTP wrapper around a model. I also quantise models where possible: moving from FP16 to INT8 with GPTQ typically halves memory and improves throughput with minimal quality loss for most tasks. I set up request queuing with priority lanes so that interactive user-facing requests are not blocked by batch jobs. Finally I add response caching for deterministic or near-deterministic queries using a semantic cache layer, which reduces both latency and cost for repeated similar inputs.

How do you approach monitoring an LLM in production?

LLM monitoring differs from classical ML monitoring because you cannot rely on a single numerical metric. I track four categories of signals. First, infrastructure metrics: latency percentiles (p50, p95, p99), error rates, token counts per request, and API spend by model and feature. Second, output quality signals: automated LLM-as-judge scores on a sample of production outputs, flagging scores below a threshold for human review, alongside user feedback like thumbs ratings. Third, data drift indicators: changes in the distribution of input query types, which can reveal a shift in how users engage with the product before it shows in quality scores. Fourth, safety signals: the rate at which the output filter or content moderation layer triggers, segmented by input category. I route all of these into a Grafana or Datadog dashboard and set alerts on metrics that have historically preceded user complaints.

Explain how you would implement and evaluate a fine-tuning run for a domain-specific task.

I start by building the training dataset carefully, because dataset quality matters more than the fine-tuning technique for most tasks. I aim for at least 500 high-quality examples in the instruction-following format the base model expects, with a 90/10 train/eval split. I use LoRA or QLoRA for parameter-efficient fine-tuning, which lets me fine-tune a 7B model on a single A100 without full-weight updates, significantly reducing compute cost. I run the fine-tune with Axolotl or the Hugging Face Trainer, tracking training loss and eval loss per epoch to catch overfitting early. After training, I evaluate the checkpoint against the held-out eval set using task-specific metrics such as F1 for classification or ROUGE-L for summarisation, and compare to the untuned baseline. I also run the checkpoint against my safety evaluation set to confirm fine-tuning has not degraded refusals on harmful inputs, which is a common side effect of domain fine-tuning.

AI Engineer

AI Engineer interviews are essentially a test of whether you've actually shipped ML in production, not just built things that worked in a notebook. Interviewers will probe across the whole stack: model selection, prompt engineering, retrieval design, deployment, monitoring, and what you did when something broke at 2am. This guide covers the questions that come up most often, and the answers that tend to land well.

For general interview preparation tips, read our guide to common interview questions.

Prepare further

How to ace the behavioural interview (STAR method)How to prepare for a technical interview

Common AI Engineer Interview Questions

Behavioural Interview Questions for AI Engineer Roles

Technical Questions for AI Engineer Candidates

What Hiring Managers Look for in AI Engineer Interviews

What hiring managers really look for in AI Engineer candidates:

Production experience over research credentials. Side projects and Kaggle notebooks don't substitute for having shipped and maintained an LLM feature under real load, and interviewers can usually tell within the first few minutes which one you have.
Full-stack understanding, not just depth in one layer. Strong candidates know how retrieval, inference, and evaluation connect, and can explain what breaks when one of them doesn't.
Cost and latency awareness from the start. AI systems that work but aren't economically viable don't last, and candidates who've thought about unit economics before being asked tend to be far more useful in product environments.
Responsible AI as a baseline, not a bonus. Safety, bias, and monitoring come up from the first round now, and candidates who treat them as afterthoughts are increasingly filtered out early.
Real curiosity about the model landscape. The field moves fast enough that candidates who can name current model families and their trade-offs signal they're working with real practice, not recycled blog posts.

Questions to Ask Your Interviewer

→What does the current AI infrastructure look like, and what are the biggest gaps you are trying to fill with this hire?
→How do you handle model versioning and rollbacks when a new model version degrades quality in production?
→What is the team's current approach to evaluating LLM output quality, and how mature is the eval tooling?
→How are decisions made about which AI use cases to invest in versus deprioritise?
→What are the biggest responsible AI or safety concerns the team is actively working on?

Practise These Questions Before Your Interview

The mock interview tool builds a practice session around a specific job posting and your background, so you rehearse the questions most likely to come up.

Start Practising

Free to start. No commitment.

Related Roles

Software Engineer

Tech10 questions

Machine Learning Engineer

Tech9 questions

Data Scientist

Tech9 questions