AI PM technical interviews test whether you can connect a product problem to the right technical choices. These six concepts help you explain how an AI system works, where it can fail and which trade-offs matter.
1. Knowledge problem or behaviour problem?
Before proposing a fix, understand what the model is missing.
A knowledge problem means the model doesn’t have the information needed to answer correctly. That information might be a recently updated policy, a private document or a customer’s current order status.
The usual approach is to supply the missing information through the prompt, retrieval or a tool that reads live data.
A behaviour problem means the model has enough information but responds in an unsuitable way. It might ignore the required format, write an overly long answer or use the wrong tone.
Start with clearer instructions, examples and output validation. Fine-tuning, which further trains a model using selected examples, may help when a repeated behaviour remains difficult to achieve through prompting. OpenAI’s model optimisation guide describes these approaches.
There’s also a third possibility: the model has the evidence and understands the instructions, but cannot reliably perform the reasoning. A stronger model or moving the calculation into ordinary code may help.
Sample response in an interview: “I’d first check whether the correct information reached the model. If it did, I’d investigate instruction-following and reasoning.”
Common misconception: Fine-tuning a model on company documents is a dependable way to keep its knowledge current. For frequently changing facts, retrieving the current source is usually easier to update and verify.
2. The context window is not memory
The context window is the limited amount of information a model can work with during an invocation. It can contain instructions, the user’s question, earlier messages, documents and tool results. Context capacity is measured in tokens, which are pieces of text rather than whole words.
A chat application can appear to remember a conversation because it supplies previous messages again. That continuity comes from how the application manages information.
Persistent memory is information retained outside the current context, such as saved preferences or conversation summaries. The application selects relevant information and brings it into context when needed. Anthropic’s context engineering guide explains this distinction.
A larger context window lets the system include more material. It doesn’t guarantee that every detail will be used correctly. Irrelevant information can distract from the answer, while additional input can increase processing cost and latency.
Putting a small document set directly into the prompt can be a sensible starting point. Retrieval becomes useful when selecting the right information is more practical than including everything.
In an interview: Explain what the system retains, what it includes in each request and how it handles outdated information.
Common misconception: A model with a large context window automatically remembers everything across sessions.
3. How RAG works, and where it fails
Retrieval-augmented generation, or RAG, finds relevant information and gives it to a model before the model answers.
A common implementation has seven steps:
Ingest: Read documents from approved sources.
Chunk: Split them into useful passages.
Embed: Represent passages as numerical vectors for similarity search.
Index: Store searchable passages and metadata.
Retrieve: Find passages relevant to the question.
Rerank, optionally: Reorder candidates using a more focused relevance assessment.
Generate: Supply selected passages to the model and ask it to answer from that evidence.
The first four steps prepare the knowledge base. Retrieval onward happens when a question arrives. Embeddings are common, but RAG can also use keyword search. Anthropic’s retrieval explanation covers these components.
When an answer is wrong, inspect where the evidence disappeared or was misused.
Permission checks also matter. A relevant document must still be authorised for that user.
Sample response in an interview: “I’d inspect the passages supplied to the model before changing the model.”
Common misconception: RAG eliminates hallucinations. It supplies evidence; the system still needs to verify that answers follow it.
4. Embeddings and hybrid search
An embedding is a list of numbers representing characteristics of a piece of content. For text search, embedding models are trained so that related meanings tend to have similar representations.
This makes semantic search possible. A search for “stop my subscription” can find a document titled “Cancel your membership,” even though the wording differs. Embeddings also support recommendations, clustering and duplicate detection. OpenAI’s embeddings guide explains these uses.
Similarity has limits. A passage can be closely related to a question while describing the wrong product or an outdated policy.
Exact identifiers create another challenge. If someone searches for error code E104, a generally similar troubleshooting article may be less useful than the document containing that exact code.
Hybrid search combines keyword search with vector search, then merges their ranked results. Keyword search helps preserve precise terms; vector search helps with different wording. Microsoft’s hybrid search overview describes this approach.
In an interview: Choose the search method based on the questions users ask, then test relevance with representative queries.
Common misconception: A high similarity score proves that a passage contains the correct answer.
5. Workflow or agent?
The distinction is about who determines the next step.
A workflow follows paths defined by the application’s designers. It can still contain AI calls, tools and conditional branches.
For example:
Receive document → extract fields → validate required fields → send exceptions for review.
An agent lets the model decide which steps or tools to use as the task develops. It might search, inspect the result and decide whether another search is necessary. Anthropic’s guide to effective agents uses this distinction.
Use a workflow when the path is predictable. Consider an agent when useful actions depend on findings you cannot reasonably anticipate in advance.
Flexibility introduces more opportunities for mistakes. In a simplified example, if ten required steps each succeed independently with 95% probability, the probability that all succeed is:
0.95¹⁰ ≈ 60%
That calculation illustrates compounding failure. It isn’t a forecast for real agents, where failures may be related and recovery steps can help.
In an interview: Explain why the task needs an agent, what it may do and when a person must approve an action.
Common misconception: Any system that calls a tool is an agent.
6. The quality, latency and cost trade-off
Every AI product needs an acceptable balance between three dimensions:
Quality: Does the system complete the task correctly and usefully?
Latency: How long does the user wait?
Cost: What does completing the task consume?
These dimensions interact, but improvements aren’t always a sacrifice. Removing irrelevant context might improve quality while reducing cost.
Other choices involve clearer trade-offs. A reranker adds processing but may improve evidence selection. A stronger model might answer more reliably while costing more. Longer outputs take more time to generate. OpenAI’s latency guide discusses these factors.
Measure the complete experience. Time to first token tells you when output begins; time to completion tells you when the useful answer is ready.
For cost, include retrieval, tool calls, retries and human review. A cheap model call can become expensive if users repeatedly retry it.
In an interview: Define the minimum acceptable quality, the user’s waiting tolerance and the budget per successful task. Compare architectures against those requirements.
Common misconception: The cheapest model produces the cheapest product. What matters is the total cost of delivering a successful outcome.
In case you’re preparing for AI PM interviews, the checkout our AI PM interview series.