From Hackathons to Company Brain: 3 attempts at an internal AI assistant

"Can you pull these numbers for me real quick?"

It’s 4 PM on a Friday. I spent the afternoon debugging a pipeline, and am getting ready for the weekend. That’s when I receive a Slack message: "can you pull these numbers for me real quick? I need it today to finalize my design. thanks!".

It was not real quick, and it never is. The query can be written in a few minutes, but to get there I need to remember the tables to use, their caveats, the time filters, and sanity-check the results... It eventually takes an hour I don’t have.

At Kikoff, we receive a lot of these requests, because nearly every decision is backed by data. Every feature ships behind an A/B test, every ad campaign is tracked with custom performance reports, and every piece of creative is measured against its own metrics. It's a culture I love being part of, and it's a big reason the company has shipped as fast as it has. But it also means the demand for numbers is endless, and any question about the business eventually routes to someone who knows where the data lives.

This wasn't just a data team problem. Engineers, data scientists, product managers... anyone who knew where to find things became an expensive search engine for everyone else. The questions are usually simple: "What's our churn rate for X?" or "What's the logic that selects users for Y?" The answers exist somewhere in our codebase, our databases, our dashboards. The hard part is actually finding it.

When the team is small, this is fine. As Kikoff grew, though, the knowledge became fragmented across tens of repositories, multiple product lines, and an ever-expanding set of features. No single person understands the full system anymore. And the people building the system, the ones with the most context, were the ones getting interrupted most.

We needed a way to let people find answers without pulling someone else away from their work. That's what started a year-long quest to build an AI assistant, with three distinct attempts.

Let your curiosity guide you

We started with a simple premise. What if anyone at Kikoff could ask a question about our data and get an immediate answer, even if they’re just curious about a topic?

That’s when the data team built a prototype for a data assistant, Tako, during a hackathon in mid-2025. Tako was a simple web app, where users would ask a question in plain English, and get an answer backed by our data models.

With this, we could democratize "data curiosity" and give PMs, marketers, and execs the ability to self-serve the same insights that previously required a data request ticket and a 2-day turnaround.

It made for a good demo. Then came our stakeholders' questions: "what’s driving the drop in conversion rate on iOS?", "what is my top performing marketing campaign this week?". Tako didn't understand the business logic or even know where to get started, and would provide confidently incorrect answers. The questions needed a few elements that Tako missed: an investigation capability—what we'd usually refer to as "agentic behavior"—and context from our different codebases, among others.

One thing was undeniable: the demand was real. People wanted to self-serve. They just needed something that could actually deliver.

Ask-an-Engineer

A few months later, the engineering team tried to address a similar concern. What if anyone at Kikoff could ask how certain processes worked without having to ask an engineer?

Ask-an-Engineer was born and developed over the course of a second hackathon. It established the foundations of future iterations of our AI assistant by introducing two ideas that would change everything.

First, instead of generating a single query and hoping for the best, it was an agent. It could autonomously decide what to do next. It could search code across seven repositories, read files, trace logic through multiple services, and synthesize a coherent answer. Ask it "how does minimum payment get calculated?" and it would follow the flow from the mobile app through the backend to the database, citing specific files along the way.

Second, it lived in Slack. No more separate web apps or login concerns, and more importantly, it met people where they were already working.

These two ideas unlocked real, daily usage, from engineers, PMs, ops, data, and more.

However, its success created new problems. People wanted to ask it more questions, for example about our data, which it had only limited access to. Additionally, the infrastructure was fragile and required frequent reboots, which prevented many users from getting answers to their questions.

The insight that changed everything

After two attempts, we had a clear picture of what worked and what didn't. Tako proved the demand for self-serve answers. Ask-an-Engineer proved that an agent in Slack could become a daily habit. But both shared the same flaw: they tried to be one thing for everyone.

The breakthrough was realizing the bot shouldn't try to know everything—it should know where to look and which tools to use depending on the question. A data question and an engineering question might look identical on the surface ("why did signups drop?"), but they require completely different approaches to answer.

That insight drove a complete redesign during our most recent hackathon. Ask-Kikoff kept the fundamentals from Ask-an-Engineer (an agent, living in Slack) but reimagined the architecture around one principle: specialize, then synthesize.

When a question comes in, Ask-Kikoff first figures out what kind of question it is. Engineering? Data? Legal? If it's unsure, it asks the user to clarify. It then activates a specialized agent with only the tools and context relevant to that domain, out of 250+ connected capabilities via our in-house MCP: code repositories, metrics databases, experiment platform, Notion documents, BI layer, and more.

As a result, it can answer a data question by querying our metrics and generating a dashboard. It can answer an engineering question by tracing code across thirteen repositories. It can cross-reference a spike in errors with a recent deployment and an experiment rollout, something that previously required a human to log into five different tools and correlate timestamps manually.

And unlike its predecessors, Ask-Kikoff uses a queue-based architecture where incoming questions are picked up by multiple independent workers in parallel, so one slow or complex question never blocks another. A background process continuously monitors for stuck or failed jobs and automatically re-queues them, while persistent conversation state means the system self-heals from any failure without anyone needing to manually intervene.

It also connects the dots that humans typically connect manually. When a data analysis reveals something interesting, it can generate a dashboard that can be shared immediately with the team, or save an investigation to Notion for future reference. Insights are now instant, and shared using the tools that the whole company uses daily.

Looking ahead

When Ask-an-Engineer was first released, anywhere from 30 to 45 questions were asked every week to one of our AI agents. Further internal communication bumped this to 75 questions per week. Since Ask-Kikoff was released less than 2 months ago, we’ve almost reached 120 questions per week.

The questions are getting harder, too. New hires use it to onboard ("how does our lending system work?"). Experienced team members push it toward edge cases ("what's the interaction between experiment X and feature Y for users in state Z?"). Every hard question we can't answer well is a signal for where to improve next.

We’re currently focusing on three main dimensions:

• Monitoring response quality so we can systematically improve it.
• Teaching the system to learn from past interactions so it gets better over time.
• Adding new integrations that turn answers into actions (dashboards, reports, alerts, scheduled analyses), without manual steps in between.

The Friday DMs still happen. Stakeholders still have urgent data requests. The difference is they now have an assistant available 24/7 that gets them unblocked in minutes. And it’s only getting better.

At Kikoff, we’re building systems that help us move faster without compromising quality, and Ask-Kikoff is just one example of how we’re rethinking how we work. If this kind of work excites you, check out our careers page at Kikoff Careers.

Ask-Kikoff was ultimately the work of a large number of Kiksters. Thank you to Julian Bacon, Nabil Barakati, David Chi, Wenzheng Chi, Cody Crunkilton, Aishwarya Gunaseelan, Jue Hou, Jason Mandel, Karthic Thangarasu, Diego Velasquez Lopez, Paul Yang, and Andy Yin for all your contributions and help in making Ask-Kikoff!

‍

About the Author

Malo leads the Data Engineering function at Kikoff, overseeing the full data platform, from data ingestion to AI adoption. He previously spent three years at Shopify as a Data Scientist. He holds two master's degrees in Industrial Engineering and Operations Research: one from UC Berkeley, where he was a Fung Excellence Scholar, and one from École des Ponts et Chaussées, a top French engineering school. Outside of work, Malo runs, lifts, plays pickleball, and loses at chess on purpose so he has an excuse to keep studying it.

‍