What It Took to Replace a Conference SaaS Workflow With AI Agents

A $43,000 question for an events team

An event organizer can accept a talk and still have a long way to go. The speaker needs a portal for a headshot and slides; organizers need reminders, room assignments and a public agenda that stays current as titles and schedules change. The same person might also be a sponsor or attendee. The hard part is making those roles and updates work together.

In Kill My SaaS, the AI Engineer events team tested whether agent-assisted builders could create an alternative to a speaker-management product quoted at $43,000 a year for four events. The host says the team used only a portion of the proposed bundle and wanted changes faster. That figure was a quote for software the team was considering, not a measured saving from a completed migration.

The organizer had long guarded against building commodity software in-house. The experiment put the opposite proposition to a practical test: give builders the jobs to be done and let the people running real events judge whether the result is useful. The video reports 72 people or teams submitted; it does not establish that an event had already run on the winning software.

The workflows that a feature list missed

Builders received a brief, a product walkthrough and 42 screenshots. The challenge offered a $10,000 first prize and up to $500 in token reimbursement for serious attempts. The event team made the final decision through a semi-blind evaluation.

Three recurring gaps showed why a screen-by-screen copy could be incomplete:

- Different portals: speakers applying through a call for papers, invited speakers and sponsors need distinct flows that an administrator can configure. - A working schedule: dragging a session should actually move it, make its status clear and avoid conflicting room or speaker assignments. - An invited talk: an organizer needs to add one without pretending that the speaker submitted it through the application form.

The host says those requirements were not well specified in the supplied material. A builder discovering them by using the product has a different task from producing a convincing page: the interactions and the rules behind them need to survive an organizer’s actual workflow.

An automated judge and the people doing the job

The team released an AI judge as a practice environment, not as the final decision-maker. It used Claude to log in as different roles, click through 20 written user scenarios and grade the results. Builders could use low scores to find areas to improve while being allowed flexibility in the labels and placement of controls.

The reported coverage narrowed at each step: 56 apps were run through the evaluation, and 40 had enough coverage to rank. The top seven scores were within 2.3 points. Those scores helped teams improve individual flows; they did not tell the organizers which product they preferred as a whole.

The divergence was substantial. The host reports that six of the automated judge’s top ten were rated weak by human reviewers, while the eventual winner ranked 19th on that leaderboard. The reviewers were looking for judgment about the job and its overall experience, including improvements outside the checklist. This is a result from this contest’s evaluators and rubric, not a universal accuracy comparison between AI and human judges.

The work before and after code generation

The finalists described starting with research and specification. One planned the data schema and business logic in detail; another converted the brief to Markdown and gathered documentation from the existing product’s help site. A third mapped product flows and asked agents to divide independent work from tasks that had to happen sequentially. These inputs gave the coding tools context about the system they were meant to build.

The agent workflow still needed feedback. One finalist found that an agent had completed its process without launching the app locally, then repeated the work with the application running. Across the finalists, the organizers say, people repeatedly used their own products and corrected what they found.

A particularly revealing bug crossed a boundary between features: an uploaded headshot was saved, but the public speaker page read from somewhere else. Each piece could appear implemented in isolation while the end-to-end user action failed. Another builder found that switching views required signing out and back in, and kept a running list of fixes after clicking through the app.

The submitted process documents also showed different ways to divide agent work. The host describes one project where one model planned and reviewed while another wrote code, and another where separate teams worked in separate worktrees and checked each other before merging.

What the team chose

The organizers said five apps crossed their bar for adoption or serious purchase consideration, and another 12 contained ideas they wanted to borrow. Open Rostrum, by Valentin de Matos, won the $10,000 prize for multiple portals and a schedule the team wanted to use. Konferencer, by Yazin, was first runner-up; its single-page view of deliverables and emails across the speaker lifecycle stood out. Marco Kremer’s Track Stage was second runner-up, with a strong drag-and-drop scheduling interface and documentation.

The host anticipated a 2027 system combining Open Rostrum, Konferencer and ideas from other entries. That was a plan at the time of the recording, rather than evidence that the mixture had already been deployed or operated successfully at a conference.

Ownership is also an operating decision

Building the initial application shifts responsibility for the ongoing service. The events team described fragmented integrations between registration, sponsor tools and the schedule, and said it needed someone who could investigate and fix problems when they occur. In one example, a sponsor could not unassign tickets; the support chain ended with a request to refresh a cache. The issue for the team was accountability for a working experience under event deadlines.

The host offered a rough estimate of $1,000 in model usage plus $10,000 in AI engineering expertise for cloning this class of software. Treat that as an estimate, not the recorded total cost of this contest or a complete lifetime cost. The owner also has to maintain the result; the opportunity is to direct that work toward the team’s own needs and respond to its users.