What Should an AI Agent Be Allowed to Write Down?
Transcription tells you what was said. The interesting problem starts one step later.
A voice recording is about the most unguarded input you can hand a piece of software. People say things in meetings they would never put in writing: what a competitor is really like to work with, why a deadline actually slipped, that someone’s co-founder is unwell and the next two weeks are going to be rough. Not indiscretion, just how humans talk when the medium is air rather than a document.
I record those conversations on a Plaud device, and I built a tool called debrief that takes one of those recordings and turns it into a brief committed to the relevant repository, plus a set of GitHub issues. Transcription itself I don’t do, because Plaud already did it. What the tool works out is what to do about what was said, checked against the roadmap, the open issues, and the conventions of the project you’re standing in.
The engineering I expected to be hard was the comprehension. It wasn’t. The genuinely hard part, the part I’ve rewritten more times than anything else, is a governance question wearing an engineering costume: what is this thing allowed to write down?
Because a brief isn’t a chat window. It’s a file, in git, with history, in a repo that might be private today and public in eighteen months.
The part I didn’t have to build
debrief reads recordings through Plaud’s MCP server rather than through exported files, and that distinction does more work than it sounds like.
Two calls carry the entire front end. get_file returns the recording’s metadata: name, timestamp, duration, and speaker count. get_transcript returns the content, and where a polished transcript exists I take that over the raw one, because it’s markedly better input. A cleaned transcript means the model spends its attention on judgment rather than on repairing dictation.
The speaker count turns out to be the most load-bearing field in that response. It selects the entire downstream shape. More than one voice and the recording is a client meeting, which gets a template built around the delta against the plan: what was already tracked, what’s new scope, what changed, what got ruled out. One voice and it’s a solo brainstorm, which gets a per-item template with a parking lot for thoughts that aren’t ready to be issues yet. Those two documents have almost nothing in common, and the choice between them is made by a single integer that arrived in the metadata for free.
That’s what I mean about transcription being solved. Not merely that the words are accurate, but that the recording arrives already structured enough to route on.
One honest footnote, since anyone who has done this will recognize it. Transcription reliably mangles domain nouns, and with some irony “Plaud” is one of them: it shows up as plowed, plod, or cloud depending on the sentence around it. So every project using this keeps a small vocabulary table of the terms its own recordings get wrong, and the brief-writer corrects against it on the way past. That table is the cheapest accuracy win in the whole setup, and it should grow every time a brief gets a term wrong.
The naive version fails in a way that gets worse as it improves
The obvious pipeline is three steps: transcribe, summarize, commit. It works on the first try, which is the problem.
The failure mode isn’t a bad summary. A bad summary is obvious and you fix it. The failure is a faithful one, which cheerfully preserves the offhand remark about a client’s payment history in a tracked file, accurately, forever.
Sit with the direction of that. Every improvement to the summarization makes it worse. A better model captures nuance more completely, which means it captures the confidence more completely too, and lands it in git history where deleting it later requires rewriting history rather than editing a file. The naive design gets more dangerous as the underlying technology gets better, which is not a property you want in something you’re going to run unattended on real client conversations.
So the interesting rules aren’t about extraction quality. They’re about restraint.
The rule that sounded responsible and wasn’t
My first attempt at restraint was the one everybody reaches for: flag anything personal.
Route it to a ⚠ Sensitive — review before commit block at the bottom of the brief, let a human decide. It sounds careful. It’s the kind of rule that reads well in a design doc.
Run it against a real recording and it does something stupid immediately. Someone mentions what they had for dinner. That’s personal. Into the sensitive block it goes, where it gets a heading, a warning symbol, and a permanent home in a tracked file. The rule designed to keep trivia out of the repo has just escorted trivia into the repo with ceremony.
And it does worse damage than the clutter suggests. A sensitive block full of somebody’s dinner is a sensitive block nobody reads carefully, which means the one line in it that genuinely mattered is now buried in chatter. An over-triggering flag doesn’t just add noise. It degrades the signal it exists to protect.
The correction is a single distinction that took embarrassingly long to see: sensitivity is the trigger, not personal-ness.
“A co-founder is unwell” is a flag. What someone ate is not, and it shouldn’t be flagged, it should be dropped on the way past, the same way filler gets dropped, without ceremony. Health, family, legal, financial, third-party confidences, unflattering characterizations: those are sensitive. Personal-but-trivial is just noise, and noise belongs nowhere, not even in the warnings.
The same rule, exactly backwards
Here’s where it got genuinely interesting, because the fix above is correct for one kind of recording and actively wrong for the other.
For a solo brainstorm, one person thinking out loud, personal-sounding talk really is noise. Drop it.
For a client meeting, the personal-sounding talk is frequently the most decision-relevant thing said all call. Capacity, absence, a reorg, budget anxiety, someone leaving: these arrive wearing the costume of small talk. “We’re slammed since the reorg” sounds like a throwaway line and is, in fact, the reason a date is about to move. A brief that drops it as chatter produces a “what changed” section that is worthless, because it records that the date moved and not why.
So the rule can’t be about the register of the remark. It has to be about relevance. Judge on relevance, not on register. Carry the substance, drop the pleasantry, and where the substance is also personal to somebody in the room, paraphrase rather than quote.
That third-party rule ended up mattering more than I expected. In a meeting the author’s own reasoning is the valuable part and worth quoting directly. Everyone else’s words are their information as much as the project’s, so they get rendered in paraphrase, and any characterization of another party’s commercial position stays out of the tracked file entirely.
The subtlest bug: flagging by quoting
This is the one I’d most want another engineer to take away, because it’s genuinely counterintuitive and I got it wrong at first.
Say the tool does everything right. It correctly identifies that the parties discussed rates and payment terms, correctly decides that doesn’t belong in a repo, and correctly routes it to the sensitive block for human review. Responsible behavior all round.
And then it writes the rates into the sensitive block, in order to flag them.
The flag is now the leak. That block is in the same file as everything else, committed by the same commit, in the same history. A quote in the sensitive section is exactly as committed as a quote anywhere else. The mechanism built to keep the content out of the repo has put the content in the repo, with a warning label on it.
The rule that fixes it is short: name the category, never quote the content.
Commercial terms were discussed and are excluded from this brief.
That’s the correct shape. The human reading it knows precisely what was left out and can go listen to the recording if they need the specifics. The specifics themselves never touch git.
A couple of categories don’t even get named that carefully. Anything the repo’s own docs designate as confidential, and anything covered by a positioning rule in the project’s CLAUDE.md, doesn’t get quoted even to be flagged. Those aren’t judgment calls the agent makes per-recording; they’re settled upstream, in the policy layer that every session reads before it does anything.
Surfaced, not silently filtered
There’s a tempting simplification lurking here, which is to have the tool quietly drop anything sensitive and say nothing.
Cleaner output. Also strictly worse, and worth being precise about why.
A silent filter means the human can’t audit the judgment. They get a brief that looks complete, with no indication that a decision was made on their behalf, which means they have no way to catch the case where the tool got the call wrong in either direction. Something important omitted looks identical to nothing important having been said.
Naming the omission keeps it a visible decision. The line “commercial terms were discussed and are excluded” is doing two jobs: it protects the content, and it tells the reviewer that a judgment happened here, so they can disagree with it. That second job is the whole point, and it’s the same principle underneath everything else I’ve written about working with agents. The agent’s call has to remain legible to the person who owns the outcome.
Three approvals that don’t compound
The structural piece is that no single “looks good” carries the whole workflow.
The brief lands by pull request. Never a direct commit to the default branch. A human reads the diff, including the sensitive block, and merges it or doesn’t.
Approving the brief is not approval to file issues. That’s a separate skill with a separate gate, and it will not create a single issue without an explicit go-ahead for that specific set. Approval doesn’t carry across runs either. Writing something down and filing work against it carry different risk, so they get different gates.
Everything created gets a triage label, without exception, however obvious the item looks. Being filed isn’t the same as being accepted.
The principle worth stealing: approval should be non-transitive. A human saying yes to one artifact must not silently authorize the next action downstream. Most of the ways agentic workflows go wrong are a single approval quietly propagating further than the person intended, and the fix isn’t more review, it’s putting the boundaries where the risk actually changes.
Raw transcripts, incidentally, never enter a repo at all. They stay in local scratch. The recording is the input, not an artifact.
Write today for the day the repo goes public
The constraint that clarified most of these decisions is a scheduling one. A private repo can be made public later. Open-sourcing a tool, handing a project to a client, publishing a case study: all of those are ordinary, desirable events, and every one of them relocates your entire git history into public view.
So the standard isn’t “is this fine in a private repo today.” It’s “does this line survive being public in two years, when nobody remembers the context and the person quoted has moved to a competitor.”
That reframing kills most of the arguments about what’s acceptable to commit, because almost nothing borderline survives it. It also means the file naming matters: a recording titled for its commercial half shouldn’t put that word in a path that lands in git history, where it’s visible in a directory listing long after anyone has read the file.
What the hard part actually was
I set out to build something that understood recordings. What I built is mostly a set of rules about restraint, wrapped around a model that was already good enough at the understanding part on day one.
That ratio keeps showing up. The capability arrives more or less free now. What you’re actually engineering is the boundary: what it may touch, what it may keep, what it must show you, and which of your approvals it’s allowed to treat as permission for something else.
The transcript was never the hard part. The commit was.