Six million bets on unreliable ground

The client reached us with a simple task:

And 4 weeks later, we were out of ideas, with the project working only partially.

The code was a zip file, no framework, all proprietary. No chosen architecture, just a happy-coding approach. Docs: 4 files, 2 PDFs and 2 txt files. One of them was a list of what queries we should run on the DB to make it work.

As you suspect, those were incomplete. The DB was just a SQL dump file, not up to date with the code; the "migrations doc" was lacking a lot of "migrations". The code didn't know what schema and migrations are.

It was a waste of time for us and the client, and every answer and step forward created a new set of problems. We couldn't estimate how much more we needed to make it work, as we simply didn't know where we were heading. And LLMs were not a thing then :)

So, we decided to cut the work on restoring the old project, and start a new one.

But this time, instead of going with a full-featured project as before, an MVP was planned.

Support for one game, from one game provider and for one operator, was scoped and we started our work.

iLotto Solutions was a lottery platform.

That means we provided lotteries for all interested clients.

Lotteries came from different sources; we offered:

• National lotteries such as EuroMillions - those are well-known lotteries; we offered betting on them and we provided their results
• Lotteries provided by other well-known providers, with insurance or not.
• Flash games (keno and such)
• Any game based on our own RNG (random number generator)

iLotto Solutions could handle the full user journey connected with purchasing a ticket - the client needed to provide basically 3 API methods for us to be able to work:

• the auth endpoint that allowed us to validate the user
• the purchase endpoint that allowed us to charge the user
• the settlement endpoint that allowed us to pay out winnings.

Initially the project was scoped to be an MVP.

And it was also agreed that, because of time constraints, we would do it quick and dirty, and once done we would have time for a proper refactor.

But reality was harsh. The lottery worked, the client wanted more, the client's client wanted more. So we started adding new lotteries and features, fighting the MVP approach whenever it was possible and we got a green light from the client.

This would backfire on us again in the future. It was one of the key lessons we learned then: sometimes you really have to stop the client, and do the boring job. A lot of things would be easier and cheaper later on.

Placing a lottery bet is trivial for the end user. Most of the time it's just a click.

• The numbers are already randomly picked.
• The next draw is selected.

Then money comes back to the wallet, or not ;), automatically. It's a frictionless operation that gives the impression of something absolutely easy. But reality is far from that, and here's what's actually happening.

There are 3 elements in the game:

• Operator - the party that has users, and holds their money
• Lottery Platform - Us - the system that provides the lotteries
• Game/Results provider - the actual entity that is responsible for doing the draws.

There are a lot of operators, fewer lottery platforms, and even fewer game providers. Each element focuses on certain aspects of "the game". So, what happens when - in the simple case.

1. The game/results provider opens the game, and the draw. Most likely one at a time, or maybe a few "in the future".
2. The lottery platform picks it up and allows placing bets on it.
3. But sometimes the operator wants to be able to place bets further into the future than the game provider offers. The lottery platform takes care of that by creating buffer draws in advance, before they are available at the game provider. Later on we sync them.

Once that happens, the user can place a bet.

• Picks up the numbers,
• Clicks OK, and hopes for the best.

The lottery platform picks up the signal. Checks all elements of the puzzle:

• is the auth data correct?
• does the user's wallet have enough money for that (we can't charge yet)
• is the draw still open?
• is the operator's API working?
• is this draw already open?
• does this draw exist yet, or is it a buffer?

A "no" to the first 3 will come back to the user as an error. If the ticket survives that, then routing happens. If the draw is not yet accepting bets (buffer draws), the system puts all actions on hold until it is able to convert the buffer draw to an actual one.

And if the draw is open -> the system creates the events that will:

• charge the user
• register the bet in our system
• register the bet at the game provider
• register the bet with the insurance (that might be covered by the game provider too)

After that, the system will wait for the draw time to come. This will trigger another set of events:

• fetch draw results
• fetch each single bet winning status (if resolution of the bet is outside of the lottery platform)
• pay out the winnings / mark losses.
• mark bets as settled in all systems.

This is for the simple bet scenario - what's not mentioned here is:

• failure recovery - any non-critical error (and we consider the API being offline a non-critical one ;)) will cause a retry of the event. The system will try to fulfill the work until it's really not possible.
• rollback routines - if an error that can't be recovered happens, a refund/rollback needs to happen.

The Editec integration was a pain.

Editec was one of our first integrations, a game provider handling very popular games in Africa. Sold until then only offline/via handheld terminals, with no web support.

• The old, overly complicated API written in C#
• The performance was so slow it needed 12h to calculate the results of a single draw.
• It ran on a machine that was, literally, under the desk of the head of IT. In our case, the cleaning lady turning it off by accident was not an anecdote. It happened.

If those 3 points above aren't enough:

It was slow; we had to buffer bets for half a day until it turned back on. It was off, because it could not handle the new bets and the settlement of old bets after the draw at once. It provided dates and times without a timezone marker, so we didn't know when daylight saving kicked in. It randomly returned empty responses. But not error-like empty responses. Just lists with no items, so it looked like it worked, but there was nothing to work on.

This was a really hard lesson for us, used to good REST APIs: a big part of the MVP was made just to cover the bugs, oddities and misbehaviours of the Editec API.

Editec was an extreme situation, but other systems were not easy to operate with either. None of the integrations we made, except for LTech, had a test/staging environment.

We were either offered "short acceptance tests" or a fake prod user account. Bearing in mind the scale of our tests, we could easily kill production by accident. What was even harder was syncing both the operators' test/prod user accounts with the game providers' test accounts; it was practically impossible, as in most cases this required a backend integration to be set up correctly.

And that was a no-go for most of them.

Frustrated by the problems with Editec, I took Node/TypeScript and a MariaDB - I created a 1-to-1 clone of their API based on the docs, and faked the logic behind it.

I copied their bugs, the ones we knew of. But also, the logic was faked only in places it needed to be to make it easier. The main logic was still proper, up to the level that we could apply the original validation rules, and keep the data integrity correct.

To be honest, the only thing that this simulator lacked was the ability to work at a big scale. It was really fun to do (knowing how much time this would save us was a big motivator), but this also showed me how easy it is to copy some systems, just by getting their docs, and how little value is in the code itself.

The simulator became the core of our work. Any new integration was first added to the simulator, then our system added support for it, within the simulator. When final acceptance tests were done, fixes were applied both to the system and to the simulator, so that it behaved exactly like its source.

The simulator not only allowed us to get around the technical instability of integrations, but also to start working on things we didn't have access to. Legal & bootstrapping time for such integrations was weeks, weeks during which we could already work, just because we were able to obtain the documentation quickly.

But it's important to know that the simulator was a part of the code that was simple: creating a new simulator module for it was a breeze.

Once we had the backend integrations ready, we thought: Why not run our frontend on it as well? And so we did; it became a tool to host different versions of our frontends, with a plethora of configurations it had. All connected to the backend integrations we had already simulated.

This became a daily tool for our testers, a demo solution, and a prototyping area.

Shortly after this, we decided that, to make it a full solution, we would add bots to it. Bots that would press buttons on the frontend and place bets. This allowed us to keep the dev/staging system busy all the time. It allowed us to see performance bottlenecks that only happened when the DB was full of historical bets, which could not be done by unit/e2e tests. Any mistake, hard to catch until then, was just popping up in log monitoring.

And from time to time the simulator misbehaved. It turned off for a few hours, started giving empty responses - just like the real one. And we could test how quickly the system recovered in such scenarios.

This happened 24/7. Regardless of us doing any work, the tests were happening all the time. The bots were trying to kill the system, the log alarms notified us if they succeeded.

One of the technical struggles we had back then was:

• How do we run our frontend (number picker) on our client's website. On any client's website.

While it sounds quite trivial now, it wasn't back then: browsers were in open conflict with iframes - <iframe> was considered a bad practice. JavaScript still had compatibility charts for different browsers.

Obviously, we could not ask our clients to include the full source code of our frontend. We had to be able to update it remotely, change it whenever we liked.

We decided to go with a JavaScript, React-based web page that would render itself under the specified tag on the client's page.

The client had to include a link to our injector script, with proper config, and this in turn ran the JavaScript that created the full DOM in the client's page.

It worked well, with some problems for the first few years that basically resolved once browser support for JavaScript matured.

Ironically, after a few years we added iframe support, as browsers decided that now, iframes are "the way".

What started as an MVP of one lottery from one provider, for one operator, with a shy 5000 bets per month, we moved to:

• ~250k bets per month 9 months after
• ~350k bets after another 8 months.

We hadn't noticed the increase at first, as autoscaling was handling it all for us, keeping the minimal number of instances through the day higher and higher. Until we ran a monthly sales report.

Our lotteries were sold in 16 African countries, spanning almost 1/2 of the continent's population, in 5 languages. We offered around 20 lottery types - each a base for per-country variations.

Initially, we offered the games provided by others. Games are regulated territory; you can't just offer them. Our next steps were to create our own RNG machine, certify it and start selling games based on our results.

This became a thing, and we could bootstrap new games within a few minutes, instead of weeks before, when a set of contracts with third parties was needed.

We handled the problems and built tooling that allowed us to work and test things. We ran this at scale. We got a stable solution that worked well for both happy paths and those not so happy.

We served 6 million bets. Out of those, some had problems:

• 4.6% of bets were cancelled. This happens on many occasions, most commonly when the draw does not take place, a bet is submitted after the draw close time, or the provider fails to register it. Our clients could sometimes buy tickets 4 weeks in advance - politics, weather, cleaning lady, all of them could stop the draw from happening. But cancellation is a normal flow. Cancellation is an established process that gives users their money back.
• 0.3% of bets were rejected. This means we didn't allow the bet to be purchased at all. This includes a lack of funds (in its edge cases) or blocked accounts. Also a normal procedure.
• Through all the years, the API was down for about 6h total. This was when we were applying big infrastructure changes.

Out of all of them, only 3000 bets were left for customer support to handle. That's 0.05% of all of them. Bearing in mind all the problems we had, that's an impressive number.

Now, let's go to the technical details. What we've used and how:

Codewise:

• API -> Built on the Symfony PHP framework.
• Additional tooling:
  • RNG -> Go
  • Simulator -> Node
  • Minor satellite services -> Go
• Queues: RabbitMQ
• Redis

Infrastructure:

• Terraform
• Containers/Docker
• AWS
• EC2, with autoscaling groups for the Docker Swarm and the RabbitMQ cluster.
• RDS
• Lambdas

So, quite a standard architecture, but a few areas are worth more than bullet points:

Swarm

That was still a thing back then. I can't say a bad word about it; we never had problems with it. Apart from one.

Swarm doesn't work well with EC2 auto scaling groups removing nodes from it: by default, it thinks that the node went offline (in an unplanned way), not that it was removed on purpose.

The solution to that was to manually remove the node from the swarm - we did that based on events provided by auto scaling.

Small Lambdas were updating swarm state accordingly.

RabbitMQ

Our second autoscaling group was dedicated to RabbitMQ.

RabbitMQ is quite picky when it comes to memory: out of the box, it sets a memory limit of 40% of what your machine has. And it will lock itself if messages take more.

That's really frustrating. Once locked, it needs to be unlocked manually (obviously, that could be automated, but you never know if that was a real problem or just a temporary spike in usage).

For most common RabbitMQ use cases that's not a common problem: typical behaviour assumes few queues with many (aka hundreds of thousands) messages. This is memory efficient, as messages are most of the time small.

In our case, we had a lot of queues with a small number of messages. A queue is much more memory-consuming, and we were eating RAM much faster. Also, our queues could hold messages for very long periods of time (weeks), and the build-up could be dramatic.

So, autoscaling was working based on the memory usage of nodes; the memory metric is not something that comes out of the box for AWS EC2. All our machines were running on containers with custom metric collection added.

Monitoring

For monitoring we decided to go with CloudWatch. The choice was mostly based on the fact that we didn't have the time and resources for EKS/Grafana stacks, and CloudWatch could give us all we needed.

We customized our Docker containers to collect custom metrics of our choice. Fluent was used to collect them, and then to proxy them to CloudWatch. Due to the unstable nature of all our integrations, we had multiple alarms configured; those were notifying us via email and Slack about any problems.

The technology behind the system was well known, established and simple. And it should be that way - it allowed us to focus on functionality.

Around 2021 our company got quite substantial funding for our startup idea: StreamSage.

At this point, we informed the client that we could no longer work on the product, as we needed all hands to work on our idea. Of course, we could not leave him on ice; the transition period took a few months, during which we found a company to replace us, but finally the client decided to hire his own devs.

We transferred the knowledge and documentation and moved on, keeping in touch with the client from time to time, giving some help on the way.

But a few months later, the client came back. An in-house dev shop is no walk in the park, and work was not going as it should; we were asked for minimal active support/development for new features.

Back in the project, we took over the implementation of new functionality required by retail/offline sales. Together with the team, we made changes to:

• the hierarchical structure of cashiers, shops and divisions - along with the reporting needed for it.
• number picker - we added features for:
  • printing physical tickets,
  • scanning QR/barcodes for settlement purposes, and
  • managing the daily chores of a lottery shop/seller.

All of this allowed the product to reach new clients.

After all these years, I think not much. The only thing I do regret is the fact that we continued with the MVP version, and never stood our ground to stop new feature development and at least rebuild it.

But! It has to be remembered that the project was under constant pressure from new clients/new features, so stopping new development at that point was practically impossible.

There are important things, and making money is always #1 in commercial code. A beautiful codebase in a dead project is worth nothing. An ugly one will slow you down when the money comes. The second one is fixable.

Through its lifetime, the project did

• 6 million bets
• About 370 lottery setups
• for 16 countries

With APIs going down for days, returning missing data or having data integrity problems, we've built a tough, resilient and failure-proof solution that did its job.

But one day I had to call terraform destroy on production.

With all things backed up, I pressed enter, and the project right now exists only on limited resources for demo purposes, ready to be turned on at any time.

All done by a small core team: me, two backend developers, one frontend developer and a tester.