High Taste Testers: Making AI Assistants Useful in Practice

The assistant that needs an open laptop

A participant wanted a new assistant to operate an existing company workflow: a Git repository of Markdown files and scripts, including one that pulled bank data for weekly updates. The assistant could not use that script through its cloud interface unless a local Codex desktop session remained open. The task sounded like “configure my assistant,” but the obstacle was where the tools and credentials actually lived.

That example opens a practical thread in High Taste Testers, a roundtable recorded after OpenAI’s Dev Day. The participants try different assistants and coding agents, disagree about trust, and compare the small conveniences and awkward handoffs that determine whether a system fits into a day.

One participant described leaving a laptop open at the office to keep an agent working. Another called remote execution and its interface a hard engineering problem. A third pointed to a possible ingredient for moving a session between machines: externalize its state in an append-only log, so another environment can resume from the recorded history. But the group also said an internal log is not enough if its format or API is not stable enough for other tools to depend on.

Fast decisions inside a product

The conversation then moves from full assistants to smaller decisions inside software. A classifier can choose among predefined outputs instead of generating a long answer: for example, rank 20 pages in a command menu from the few letters a user typed. The panel’s point is that lower latency and cost can make a model practical in an interface where waiting for streamed text would be awkward. Structured output and classification themselves were already possible.

A number attached to a decision is not automatically a reliable probability. Participants contrast model-produced scores that bunch around a few values with the prospect of better-calibrated decision probabilities. They also caution that a binary answer can hide a hard problem: deciding whether a statement is true may look like a yes-or-no classification, but the difficulty is in knowing the answer.

One participant suggests reusing historical labels as an evaluation when a cheaper model appears: compare its decisions to the old labels and inspect disagreements. That is a useful migration test, especially where the task is stable. Agreement alone only establishes similarity to the old labels; it cannot establish that those labels were correct, so checking a sample of agreements as well would help detect shared errors.

The panel offers a more involved example: an agent-memory pipeline alternates a fast decision model for sorting and reranking with a larger model, then proposes what to add to a project’s instruction file from agent traces. One participant reports a 10–20× cost reduction and says its evaluations aligned reasonably with a larger model. The recording provides limited detail about the dataset and evaluation.

More parallel work, more to read

One participant says running models overnight on production software can generate more code than a person can reliably absorb and review. Another account describes running ten prototypes overnight, with a bounded queue of five to ten prompts that asks the agent to revise, compact context, consult the plan and continue. The queue provides a stopping point; it does not remove the need to inspect what came back.

Others describe different arrangements: run multiple agents in the background and monitor logs; keep a main thread focused on delegating and reviewing; or do the work in the main thread while holding the conversation in a side thread. Their common problem is maintaining enough context to notice when a long-running task has gone off course.

One person praises a general computer sandbox for exploring an idea across repositories or outside a repository entirely. Another finds a coding-oriented agent tends to turn a request into a pull request even when an answer would do. The distinction matters at the handoff: a good tool must understand whether the requested result is a software change, an investigation, or a short answer.

Reading is a different permission from sending

The sharpest disagreement concerns an assistant with access to private information. One participant argues that prompt injection is largely solved, citing an unsuccessful attempt to break into an agent. Others dispute the conclusion and point to a separate failure: even without a malicious attacker, a helpful assistant might disclose information to the wrong person—for example, telling a boss that its user is interviewing elsewhere. A few failed attacks do not establish that all attacks fail, and a privacy mistake need not involve an attack at all.

A participant who connects email describes requiring a human to see the draft and press send. That person reports that roughly one in five drafts contains details they do not want included. Another describes a read-only default for sensitive connections and explicit permission for writes, while still wanting a way to create separate virtual cards.

A shared connections layer is one attempt to reduce setup work: one participant keeps tools, configuration and instructions in a central place used across devices, while retaining some provider-specific connectors. The tradeoff is that a connection powerful enough to be convenient can expose sensitive information across more workflows. The group also describes a mundane consequence of multiple assistants: it becomes hard to remember which one knows what, because their histories diverge.

A useful assistant starts with a specific job

A participant describes taking children to a karate class only to discover it was closed. In response, they built a weekend-planning bot to search local places, suggest activities and flag closures affecting planned outings. Its appeal is concrete: a warning before leaving home saves a trip.

The group does not settle on one interface. Some prefer a fast search answer for a quick question; another uses an assistant for short travel questions; someone else argues that splitting queries among products fragments useful context. Their disagreement shows why convenience depends on more than model output: the system must have the right information, reach the right tools, and make its proposed actions understandable at the moment the user needs them.