Directors on Directors: A Search Engine for Filmmaker Conversations
Variety’s Directors on Directors is one of my favorite things on the internet — two filmmakers, no moderator pretending to be neutral, just genuine curiosity about how the other one works. Years of these conversations exist, full of directors talking about actors, improvisation, editing, rehearsal, storyboarding, pressure, failure, taste. All of it scattered across hours of video that nobody has time to rewatch.
So I built a question-answering app over the archive. Not a keyword search — something you could actually interrogate:
“Which directors talk about improvisation?” “How do Ari Aster and Rian Johnson think differently about storyboarding?” “What happens if a question assumes something the corpus doesn’t actually support?”
The idea was fun enough to justify building it. Whether it actually worked — whether the answers could be trusted — turned out to be the more interesting problem.
Part 1 — Building the retrieval system
The first version was a straightforward RAG pipeline: YouTube transcripts as the primary source, Wikipedia pages for background context like filmography and biography, Gemini embeddings, ChromaDB, MMR retrieval, and Gemini for answer generation with citations.
The product decision that mattered most was making citations non-negotiable. An answer about what a director believes is only useful if you can see exactly which conversation it came from.
That decision is also what exposed the real problem. A lot of directors talk about the same themes — improvisation, actors, rehearsals, test screenings — so the system could retrieve something semantically similar and still be wrong. In this corpus, good retrieval wasn’t just about finding the right topic. It was about finding the right director, the right conversation, and sometimes all the relevant conversations at once.
Part 2 — Finding out if the answers could be trusted
Once the app worked end to end, I needed to know how much to trust it.
I started with RAGAS on 14 AI-generated answerable questions:
| Metric | Score |
|---|---|
| Faithfulness | 0.979 |
| Answer relevancy | 0.810 |
| Context recall | 0.750 |
| Context precision | 0.482 |
Faithfulness was strong. Retrieval clearly wasn’t. That tracked with what I was seeing manually — direct lookup questions worked fine, but comparison questions (“which directors prefer X,” “how do these two differ on Y”) needed broader retrieval, and the system wasn’t finding enough of the right sources.
So I built a manual diagnostic set targeting the failure modes I actually cared about: comparisons, false premises, missing topics, attribution mistakes, wrong-conversation retrieval.
The failures were subtle, which was the interesting part. The system wasn’t hallucinating from nothing — it was confidently using nearby-but-wrong evidence. In one case, a question about Adam McKay and Tyler Perry on improvisation pulled in a Judd Apatow anecdote, because it was topically close enough. The citation was real. It was just from the wrong conversation.
That’s the failure mode that matters most in a system like this: a RAG answer can cite its sources and still be wrong, because the source is from the wrong context.
What I changed
First pass was prompt-level — I made answer generation stricter about disclosing incomplete evidence, separating transcript evidence from Wikipedia background, avoiding unsupported attribution, and rejecting false premises outright. This helped. Regression probes passed cleanly, and the system got noticeably better at admitting when the corpus didn’t support a question.
But the McKay/Perry case pointed at something deeper. The model stopped misattributing the Apatow anecdote to McKay — but Apatow was still showing up in the answer at all. The prompt was better. Retrieval was still handing it bad evidence.
So the real fix was at the retrieval layer: participant-downranking. When a question names specific directors, the retriever now pulls a wider candidate pool and downranks transcript chunks from conversations that don’t include those directors. Wikipedia chunks stay untouched, since they’re still useful for background.
It’s a lightweight fix — no index rebuild, no extra LLM call. It just taught retrieval that “topically similar” and “valid evidence” aren’t the same thing. After this change, the McKay/Perry answer came back correct, the Apatow chunks dropped out of the answer path, and the system kept correctly rejecting false premises — like a question that assumed Spike Lee directed Nomadland, correctly redirecting to Chloé Zhao.
Final result
I expanded the diagnostic set to 49 targeted questions.
The biggest gains were on attribution and false-premise handling — the system got much less likely to confidently answer something it shouldn’t, or mix up who said what. What remained was a clear pattern: broad comparison questions where it found some correct directors but missed others.
That gave me a roadmap rather than a dead end:
- query-adaptive retrieval depth
- larger candidate pools for comparison questions
- balancing results across directors
- reranking for broad synthesis questions
- deterministic source-coverage checks
- better detection of wrong-topic bleed
What I learned
The interesting part of this project was never really the RAG app itself. It was realizing that different kinds of questions need different retrieval behavior:
A direct lookup question needs precision. A comparison question needs breadth. A false-premise question needs restraint. A research assistant needs to know when its own evidence is incomplete.
What started as an excuse to ask better questions of a messy, wonderful archive turned into a small case study in what it actually takes to make an AI system trustworthy — not just accurate on the easy cases, but honest about the hard ones.
Closing the loop with live feedback
A 49-question diagnostic set is enough to catch known failure modes. It’s not enough to catch the ones I haven’t thought of. So the app has a lightweight review widget on every answer — spot on, partly right, off the mark, plus an optional note on what was right or wrong.
