I wanted to see if I could predict a UFC fight

The first thing I needed was data, so I found a UFC dataset on Kaggle and started looking through it. I wasn't trying to build the model immediately. I wanted to see what information was actually available, how much historical data there was, and whether I could find fights I remembered and see how those fights were represented in the dataset.

That exercise raised a question pretty quickly. How much do you actually know about a UFC fighter if all you have is UFC data?

Israel Adesanya is a good example. By the time he entered the UFC, he had already had a long professional kickboxing career, but a UFC-only dataset sees his first UFC fight and effectively says that he has no UFC experience. That doesn't make the data wrong. It just means the model has a very specific definition of experience, and that definition doesn't necessarily match what I would think of as a fighter's actual experience.

I eventually moved away from the Kaggle dataset and used a deeper UFCStats-based source as the historical foundation for the project. It gave me a much better foundation for reconstructing fighter histories, but it didn't solve that fundamental limitation. The model would still be forecasting UFC fights using UFC history, not someone's entire combat sports career.

That ended up being one of the recurring themes of the project. Having a lot of data isn't necessarily the same thing as having the right data.

Once I had the data, I started thinking about the things I would normally consider when watching a fight. Age seemed important. Reach probably mattered. Experience obviously mattered, although I wasn't sure how useful UFC experience would be for someone who had spent most of their career fighting somewhere else. Striking, takedowns, defense, opponent quality and recent performance all seemed like reasonable things to consider.

The problem was figuring out which of those things actually mattered and how much they mattered relative to everything else. I could come up with a long list of things that sounded plausible, but that didn't mean the historical data would agree with me.

That was where logistic regression became useful. I've used regression analysis at work to investigate questions like whether sales activities are related to pipeline creation or whether different business decisions are associated with changes in revenue or churn, so the general idea wasn't completely foreign to me. Applying the same kind of thinking to a UFC fight was new.

The question was fairly straightforward. If I give the model two fighters and the information that would have been available before their fight, can it estimate the probability that Fighter A wins?

We chose the variables we thought were worth testing, and the regression learned the coefficients from historical fights. Those coefficients became the weights the model used when calculating the probability. I didn't decide that age should count for 20% and takedowns should count for 15%. The model estimated those relationships from the historical fights.

There was another problem I had to learn to think about much more carefully: leakage. I understood the basic idea, but I hadn't really thought about how easily it could creep into something like this when you're reconstructing what a fighter knew before each historical fight.

If I'm trying to predict a fight that happened in 2018, I can only use information that would have been available before that fight. I can't calculate a fighter's career statistics using fights from 2019 and then pretend those statistics were known in 2018, even if doing that produces a better-looking model.

That sounds obvious until you start building historical features. Take Elo ratings. If I calculate a fighter's current Elo rating and use it to predict an old fight, I've given the model information from fights that happened after the fight I'm trying to predict. The same problem can happen with recent performance, opponent quality and other statistics if you don't carefully reconstruct what was known at each point in time.

One of the stranger examples came from the early UFC tournaments where fighters could have multiple fights on the same night. A fighter's second fight of the night can legitimately use information from their first fight, but it can't use information from a fight that hasn't happened yet.

I initially pushed back a little on whether this was worth worrying about because modern UFC events don't work that way anymore. But that wasn't really the point. If we're testing a model historically, the model has to behave as if it were standing at that exact moment in time, regardless of whether the format still exists today.

That was probably one of the first things I learned that I hadn't really considered before starting the project. You can have the right variables and still build a bad forecast if you let those variables see information they wouldn't have known yet.

I wanted the first model to be relatively simple. There are an almost unlimited number of things you could measure about a fighter, and it would have been easy to keep adding features, but I didn't think that would teach me much about which things were actually useful.

The model ended up using a combination of age, Elo rating, striking performance, takedowns, defense, reach, UFC experience and opponent quality, along with a few other historical measures. Each feature was calculated using information available before the fight, and we used regularized logistic regression to estimate the coefficients.

The first question was whether the model could do better than chance. A 50/50 model is a pretty low bar, but it gives you somewhere to start, and Elo by itself did better than that.

The logistic model performed better because it could combine multiple pieces of information rather than relying on a single rating. Over our 2015 through 2024 walk-forward evaluation, Model v1 reached 61.7% accuracy with a log loss of 0.6521.

I didn't want to stop there because historical performance can be very easy to overstate if you're not careful about how you test it. We set aside 2025 and 2026 as an untouched holdout, which meant the model couldn't use those fights while we were developing it or deciding which changes to keep.

When we finally evaluated Model v1 against those 940 fights, it produced 66.0% accuracy and a 0.6297 log loss. That doesn't mean the model will predict exactly 66% of future UFC fights correctly. It means that on this particular period that we deliberately kept out of development, those were the results.

That distinction became important as I started thinking more seriously about what a probability actually means. If the model says 70%, I want that number to tell me something about the likelihood of the outcome, not simply be another way of saying that the model picked one fighter over the other.

It would have been easy to build a model that simply said which fighter it thought would win. If Fighter A had a probability above 50%, pick Fighter A, and if Fighter B had the higher probability, pick Fighter B.

But that wasn't really what I wanted. If the model says Fighter A has a 51% chance of winning, that's a very different statement from saying Fighter A has a 90% chance of winning, even though both would result in the same pick.

That led me into a much deeper look at how forecasting models are evaluated. We looked at accuracy, AUC, Brier score, log loss and calibration, and I had to spend some time understanding what each of those measures was actually telling me rather than just treating them as numbers that belonged in a results table.

Calibration was particularly interesting because it gets at a simple question: when the model says 70%, does something that looks like a 70% forecast actually happen about 70% of the time? If you give 100 fighters a 70% probability and only 55 of them win, the model might still have decent accuracy, but the probabilities clearly aren't telling you what you thought they were telling you.

That changed how I thought about the project. I wasn't just trying to build something that could pick fights. I wanted to build something where the number attached to the pick actually meant something.

Once we had Model v1, the temptation was to keep improving it. There were plenty of ideas, and because AI makes it relatively easy to test another idea, it would have been very easy for the project to turn into an endless search for another slightly better version of the model.

One of the first things I wondered about was age. A five-year age difference probably doesn't mean exactly the same thing at every point in a fighter's career. The difference between 25 and 30 might not be equivalent to the difference between 35 and 40, so we tested a nonlinear age relationship to see whether the model was being too simplistic.

It performed slightly better, but the improvement wasn't large enough to meet the threshold we had established before running the experiment. We didn't use it.

We then looked at whether Elo should behave differently for fighters with very little UFC experience. The idea was that a rating might need to move more aggressively when there isn't much history behind it and settle down later in a fighter's career. That didn't improve the model either.

We also tested a Glicko-style rating uncertainty concept, different levels of regularization and probability calibration. Some of those versions produced slightly different calibration results, but none produced enough improvement in the overall forecasting metric to justify changing Model v1.

One of the more interesting ideas was trying to account for professional experience outside the UFC. I thought this could address one of the obvious weaknesses of a UFC-only model, but the data wasn't good enough to do it properly. We could find someone's professional record today, but that wasn't enough because we needed to know what their experience was at the time of each historical fight.

That was a useful stopping point for that particular idea. Sometimes the problem isn't that the model needs to be more sophisticated. Sometimes you simply don't have the data required to answer the question properly.

The age experiment stuck with me because it came directly from something I was noticing while looking at the forecasts. I kept thinking about the difference between someone who is 25 and someone who is 30, and then comparing that with someone who is 35 and someone who is 40. A five-year difference didn't seem like it should necessarily have the same meaning across the entire age range.

The original model treated age as a linear relationship, which meant the effect of a five-year difference was basically the same regardless of where those five years occurred. We built another version that allowed the relationship to change across different parts of the age range.

The new version did slightly better across the 2015 through 2024 walk-forward period and improved in eight of the ten individual years. But the improvement was still only about 0.0008 in log loss, which was below the threshold we had set before running the test.

So Model v1 stayed as it was. I liked that outcome because it forced me to separate “this seems like a better model” from “we have enough evidence to change the model,” and those aren't the same thing.

It's very easy to keep tweaking something until you find a version that looks better on historical data. At some point you have to decide how much improvement is enough to justify the additional complexity, and for this version of the project, the answer was that the improvement wasn't enough.

Once I was comfortable stopping the model research, we could finally build the thing I had wanted from the beginning. The forecasting engine takes two fighters, reconstructs the information that would have been available before their fight, creates the Model v1 features and runs them through the frozen coefficients.

The website then turns that into something a person can actually use. You select two fighters, confirm the fight context and get a probability for each fighter, along with the factors that contributed most to the result.

I was pretty deliberate about the explanation piece because I didn't want another AI model looking at the forecast and making up a nice-sounding explanation for why one fighter was going to win. I wanted the explanation to come directly from the actual model.

If the model is giving someone an advantage because of age, takedown rate and Elo, that's what the tool should show. It shouldn't tell you that the fighter “looks more athletic” or “has better championship composure” because those aren't things the model knows.

The model should explain itself using the information it actually used. If age, takedown rate and Elo are driving the forecast, those are the things I want the person using the tool to see.

One of the tests we ran involved Ernesta Kareckaitė and Melissa Gatto. The website wouldn't let me run the matchup because it thought Kareckaitė was a male fighter.

At first, that looked like a model problem, but it wasn't. UFCStats doesn't explicitly provide fighter gender, so we were inferring it from the bout description, and that broke down for some catchweight fights because the description didn't always identify them as women's bouts.

The forecasting engine itself could handle the matchup. The problem was in the website's fighter list, which had taken the gender and division from the fighter's most recent bout without accounting for catchweight fights.

We changed the lookup logic so it uses the most recent named division instead of treating a catchweight fight as evidence of gender. The model itself didn't need to change.

It was a pretty mundane bug, but I actually liked finding it because it felt very familiar to me as someone who works in analytics. You can spend weeks thinking about model coefficients and validation strategies and then discover that the website thinks a fighter is male because of how one type of fight was labeled.

There is one distinction that became particularly important once we started turning the research project into a real forecasting tool. Model v1 is frozen because we deliberately kept 2025 and 2026 out of the development process so we could use those fights as an untouched benchmark.

That doesn't mean the fighter data should stay frozen in 2024. If a fighter has another fight, their age changes, their UFC experience changes, their recent performance changes, their Elo changes and their statistics change.

The idea is to keep the model coefficients fixed while updating the fighter histories that feed into those coefficients. That's why the website shows two different dates, one for how current the fighter data is and another for when the model was trained.

Those are two different things, and keeping them separate will become increasingly important if this turns into something I actually maintain. I don't want updating the fighter data to accidentally become an excuse to keep changing the model.

One of the reasons I wanted to build this in the first place was because I don't completely trust my own intuition about fights. I have opinions about fighters and I've watched enough UFC to have a general sense of why I think someone should win, but that doesn't mean those opinions are unbiased.

Take Justin Gaethje and Ilia Topuria. If I were making a prediction based purely on intuition, I could easily talk myself into Topuria because he's younger, in his prime, and had beaten some of the best fighters in the division.

But then there are other things to consider. Gaethje has spent years fighting at the highest level, he's physically bigger than some of the fighters Topuria had faced, and he's fought a long list of elite opponents.

Which of those things should matter more? I don't really want to answer that based on whichever argument sounds better to me on a Saturday night. I wanted something I could put next to my intuition and ask whether the data agreed with me, even if the answer wasn't always what I expected.

That's really what I wanted from the model. It wasn't supposed to replace my judgment or tell me that the model was always right. I wanted another way of looking at the same question, and sometimes the fact that the model disagreed with my intuition was probably more useful than if it had simply confirmed what I already believed.

A lot of this project was built with AI. Claude wrote a significant amount of the Python, modeling code, testing framework and forecasting engine, and it also helped build the website. I used ChatGPT alongside it to review the work, explain things I didn't understand, think through experiments, interpret the results and challenge some of the directions the project could have taken.

I'm not going to pretend I could sit down tomorrow and reproduce the entire Python project from memory. I couldn't, and I don't think there's any reason to pretend otherwise. I know SQL well and I understand the analytical reasoning behind the project, but I'm still learning Python and some of the statistical techniques that went into the implementation.

At the same time, I don't think it's accurate to say that AI did the analysis for me. I was still deciding what I wanted to investigate, what questions were worth asking, which features made sense, whether the results passed the gut check and when we had done enough work to actually build the product.

That last part mattered more than I expected. Claude could always suggest another feature, another experiment or another way to make the model more complicated, and eventually I had to decide whether any of those things were actually worth doing.

At some point we had a model that worked reasonably well, several experiments that hadn't produced enough improvement to justify changing it, and a forecasting engine that reproduced the model's historical predictions exactly. I could have kept looking for another small improvement, but that would have taken me further away from the thing I actually wanted to build.

The obvious answer is a UFC forecasting tool, but the more interesting part for me is that I finally got to build something I'd been thinking about for years. I didn't suddenly become a machine learning engineer while working on it, and I still couldn't build every part of the project from scratch without help.

What changed was that I no longer needed to know every part of the technical stack before I could start. I could bring the question, the domain knowledge and the analytical thinking, and use AI to help me work through the parts of the implementation that were outside my current skill set.

That doesn't mean technical skills stop mattering. I still think you need to understand enough of the code to question it, enough statistics to understand what the model is actually telling you, and enough judgment to know when an answer doesn't make sense.

But if you've had an idea sitting in the back of your head because you thought you didn't have the technical skills to build it, I think the cost of finding out whether the idea is actually worth pursuing has gotten a lot lower. I've had this UFC idea for years, and eventually I decided that I didn't need to wait until I knew everything to see what would happen if I actually built it.