I tried to forecast a business I built from scratch

The question I really wanted to answer was simple to state. If you understand how a sales business actually works, do you forecast it better than if you just take last year's numbers and extend them? And underneath that, if you use the forecast to make a decision, like adding more reps, do you make a better one?

The problem with answering that using real data is that you only ever get to see one future. If a company hires ten more salespeople and revenue grows, you never find out what would have happened if it hadn't. A simulation fixes that. We built a fictional software company with about 50 sales reps, about 5,000 customers across three segments, and annual contracts that renew or lapse each year. Then we ran 200 versions of its 2025 and 2026, all following the same rules, with the dice falling differently in each one, and in every version we could see what would have happened under four different hiring plans.

I want to be clear that none of this is real. The company, the customers, the reps and every dollar figure in this piece are synthetic. The questions come from my own experience working with sales organizations, but the answers only describe a business we invented. That turns out to matter quite a lot, because we also had to write down the rules of how that business behaves, and some of what we “found” later was something we had put there ourselves.

The first surprise came before any forecasting at all. To make the history realistic, we deliberately gave the raw data a few of the problems I've seen in real systems, and then had to decide how to fix them. Each fix forced a definition.

Contracts had both a signing date and a start date, and enterprise customers could take up to three months to go live. If you count recurring revenue from the signing date, you count money the company isn't earning yet, and then you count it again when the customer actually starts. Cancellations had both a notice date and a last day of service, and customers gave 30 to 90 days' notice. If you count the loss when they give notice, you book churn a quarter early. And about 2% of renewals had been entered under a brand-new customer record, so the original customer looked like they'd left and a new one appeared on the same day. None of these is exotic, and getting each one wrong moved real numbers. Counting churn by notice date overstated fourth-quarter churn by about 14%, for example.

Even something as ordinary as seasonality turned into a definition problem. We wanted the business to have a busy fourth quarter, so we checked how much of the year's new business landed in Q4. Measured by when deals were signed, it was about 28%. Measured by when customers went live, it was about 24%, because deals signed in December don't start until January. It was the same business with the same deals, and the answer changed depending on what we meant by “new business.”

This is what I mean when I say the data doesn't exist until you decide what it means. Once we had made all of those decisions, the cleaned history matched the simulation's own records exactly. Before we made them, the same database gave us different answers depending on which column you happened to trust.

With a clean history up to the end of 2024, we built two kinds of forecast for 2025 and 2026. The first, which we called Forecast A, is the kind of thing a finance team might put together quickly. It takes the amount of recurring revenue the company added last year and assumes it adds the same amount again, spread across the quarters in the usual seasonal pattern. We also built a version, A-g, that grows that amount at last year's growth rate, so the simple approach wasn't being set up to lose.

The second, Forecast B, builds the forecast up from how the business works. It starts with the customers the company already has and estimates how many will renew, and at what price. It adds the deals already signed but not yet live, and the ones still open in the pipeline. Then it estimates new business from the sales team itself: how many reps there are, how long new hires take to get up to speed, how many opportunities a fully ramped rep creates in a month, how many of those turn into wins, and how big a typical deal is. Every one of those inputs was estimated from the history the company would have had on December 31, 2024, and nothing from later.

I should be honest about how this was built. AI wrote nearly all of the Python and a lot of the SQL. I'm comfortable with SQL and with the business side of this, but I'm still learning Python and statistical modeling, and I didn't write or fully understand every line of the code. What I did was decide what we were trying to answer and make the calls along the way. Before we scored anything, I decided we wouldn't declare a winning threshold in advance. Once the forecasts were done, I decided they would be frozen so nobody could quietly adjust them after seeing the results.

When we compared the forecasts against what actually happened across the 200 versions of the future, the detailed forecast was more accurate overall, but not by much. Its typical quarterly revenue error was about 2.1%, against 2.4% for the simple forecast, and it was closer in about 61% of the futures. That is a consistent difference, but on roughly $35 million of quarterly revenue it amounts to about $100,000 a quarter. For forecasting total revenue, a spreadsheet run-rate came surprisingly close to a much more elaborate model.

The more interesting part was when each forecast did well. In 2025, the simple forecast was actually better, beating the detailed one on annual revenue in about 61% of futures. In 2026 the order flipped, and on new recurring revenue the detailed forecast's error was about 10%, against about 19% for the simple one. The reason makes sense once you see it. The company planned to grow its sales team from 50 to 58 people, and new hires take months to ramp, so that growth mostly showed up in the second year. A forecast that repeats last year can't see it coming, while a forecast built on headcount can.

The growth-adjusted version, A-g, made things worse rather than better. Last year's growth rate bounced around so much from one future to the next that extending it was riskier than not extending it at all. If I had to summarize this part, it would be that understanding the drivers helped, but mostly when the business was about to change in a way the recent past didn't show.

There's an important caveat here. Our detailed forecast is built the same way our simulated business works, because we designed both. In our company, headcount really does drive opportunities, because we wrote it that way. Some of the detailed forecast's second-year advantage might simply be a home-field advantage in a world built the way it thinks. We had planned a test for this and never ran it, so I'd treat this as a result about this business, not a rule about forecasting.

The detailed forecast's 2025 numbers were too low, and its uncertainty ranges were badly off. The forecast gives a range it expects the actual result to fall inside about 80% of the time, and for 2025 revenue the actual result landed inside that range only about 23% of the time, almost always above it. So the forecast wasn't just a little wrong. It was confidently wrong in one direction.

The cause turned out to be one piece of logic. The forecast's renewal rates came from history, and in that history every customer who left had given notice before leaving. The forecast then also took every customer who had already given notice by December 31 and marked them as certain to leave, which sounds sensible. But it kept applying the full historical churn rate to everyone else as well. For contracts ending in January and February, almost everyone who was going to leave had already said so, which meant the remaining customers were nearly all staying, and the forecast still assumed 10 to 20% of them would go. It had counted the same lost customers twice. That one mistake explained essentially the whole of the forecast's error in its existing customer base, about $2.5 million of recurring revenue.

What struck me was where we didn't find the problem. We later ran a sensitivity analysis to see which assumptions moved the forecast the most, and the answer was what you'd expect: renewal rates, price increases at renewal, rep productivity and win rates. The renewal rate alone could swing the year-end forecast by almost $13 million. But when we went back and compared each estimate with what actually happened, those powerful assumptions had been estimated almost perfectly. The renewal rate was within half a percentage point. The errors that actually hurt came from modeling choices: the double-counted churn, assuming deal sizes would stay flat when they had been growing about 4% a year, and assuming every new rep would perform like the average one.

That distinction is probably the thing I'll carry into my day-to-day work. When a forecast misses, the instinct is to go back and scrutinize the biggest assumptions, and here that would have led us to recheck numbers that were already right. Knowing which assumption matters most is a different question from knowing which one is wrong.

The whole point of building a driver-based forecast was to answer a question a simple forecast can't: what happens if we hire more people? So we asked the detailed forecast to compare four options for 2025 and 2026, keeping the planned team or adding 10%, 20% or 30% more reps. Then we judged each option on the extra revenue it brought in, minus the cost of the extra reps at $180,000 a year each.

The forecast recommended adding 30% in every single one of the 200 futures. With hindsight, not hiring was the best choice in about 72% of them. Measured against whichever option turned out best in each future, the forecast's choice gave up a median of about $1.5 million. The quick back-of-the-envelope version (“each rep brings in about $500,000 of new business a year, so more reps means more revenue”) made exactly the same choice.

The reason took me a while to really understand. The forecast estimated the average rep correctly. But the hiring decision doesn't depend on the average rep, it depends on the extra one. In our simulated company, reps share a limited pool of prospects. When we added reps, the new people produced close to a normal pipeline, about 94% of the average, but the existing team lost roughly a third of that amount, because some of those prospects would otherwise have gone to them. Each additional unit of sales capacity added only about 62% of what an average rep produces. The forecast assumed 100%.

After all of this, we built a revised forecast, B v2. I want to be upfront that it was built after we had seen exactly where the first version went wrong, so it isn't a clean test of what we would have known in advance. It's a test of whether fixing the problems we found would help.

The fix for the double-counted churn made a big difference. It estimates each customer's chance of renewing given that they hadn't given notice by the cutoff, which uses only information the company would have had at the time. We also checked it against earlier years of history. Standing at the end of 2022, it predicted renewal rates of about 99.8% and 94.6% for contracts ending in the next two months, where the real figures were about 100% and 95%, and the end of 2023 looked almost identical. The original logic said about 89% in both cases. With that one fix, the typical quarterly revenue error fell from 2.1% to 1.1%, and the forecast's ranges went from containing the actual result 43% of the time to about 77%. We also added the 4% annual growth in deal sizes, which is a reasonable correction, but on its own it didn't make the forecast measurably more accurate.

Here's the part that surprised me. With both of those fixes, the forecast was the most accurate one we had, and it still recommended adding 30% more reps in every future, with exactly the same cost in hindsight. A better revenue forecast had done nothing for the hiring decision, because the decision depended on the extra rep and neither fix touched that.

The obvious next step was to estimate how much new reps cannibalize the existing team from the historical data. We tried, and the history couldn't tell us. Our best estimate of that effect came with so much uncertainty that in about 70% of the futures we couldn't rule out that it was zero. Over a two-year window, hiring only paid off if the effect was below a certain level, and that level sat right in the middle of the range the data allowed. So instead of one recommendation, the revised forecast showed how the answer changes depending on what you believe about that overlap. A more cautious way of choosing, which only hires if it still pays off when the effect is large, would have done much better in hindsight. But we designed that rule after we'd already seen that hiring rarely paid, so I don't count it as evidence that it works.

It's tempting to read this as “don't hire more salespeople,” and that would be the wrong conclusion. We put the cannibalization effect into the simulation ourselves, so the original forecast's hiring mistake was partly built in by design. We also only judged the decision over two years. The new reps kept producing about $360,000 to $370,000 of recurring revenue each, and that revenue carries on after 2026. Our best estimate is that the hires would typically have paid for themselves after about 27 to 31 months, just outside the window we measured, so over a longer horizon the answer could easily flip. And our “benefit” was revenue minus the cost of the reps, not profit, because we didn't have a defensible way to estimate the cost of serving the new customers.

What I do think it shows is that forecasting revenue and making a hiring decision are related problems, but they are not the same problem. You can get much better at the first without getting any better at the second, because the decision can hinge on a different assumption than the forecast does. And that assumption can be the one your data is least able to tell you anything about.

I started this project thinking it was about whether a detailed forecast beats a simple one. It turned out the answer to that was modest and a little boring: the detailed one did somewhat better, mostly when the business was changing. The more useful lessons were about everything around the forecast.

The first was that business data is a pile of decisions. Revenue, churn, customers and productivity all depend on definitions that someone chose, and building a company from scratch made me choose every one of them. The second was that the most expensive mistake wasn't a bad estimate of anything. It was a piece of logic that looked perfectly reasonable until we checked it against what happened. The third was that a forecast can be accurate and still lead you to the wrong decision, if the decision depends on something the forecast quietly assumed.

If I had to bring one thing from this back to a real planning conversation, it would be to ask a different question about a forecast. Instead of only asking how confident we are in the number, I'd ask what we would have to believe for each option to make sense, and whether our data can actually tell us if that's true. In this project, the honest answer to whether the company should hire was that it depends on something we couldn't measure. That's less satisfying than a single recommendation, but I think it's closer to the truth.