Those who’ve watched The Big Lebowski will remember Walter (played by John Goodman). Actually, Thomas H. Grohser, Senior Database Engineer at bwin looks something like Walter, what I really appreciated.
Here are some facts about the bwin databases:
- 2000 bets per second
- 400 webserver
- largest DB: 5 TB
- total data: 100 TB
- 450.000 queries per second
- 170 different applications, >= 1 new release per day
Sounds impressive, doesn’t it? bwin has a mission, numbers are priority:
- zero data loss
- availability of 99.99 %
- 450.000 queries per second (with that, they’re worlds’ # 2)
Since foundation, they didn’t loose a single row, that’s for almost 10 years now. bwin can go back to every particular state, within the last seven years. In fact, availability is between 99.992 and 99.993, which means about three to five hours downtime per year, where two hours are planned and used for maintenance and updates.
How to achieve that? Standards, massiv redundancy and “Work by the book” are key factors. Every server has the same hardware. And if there appears any error on administrators’ eight wide-screen TFTs, there are strict orders what to do. Of course administrators are redundant too, it’s almost like on an air plane.
bwin does backups, constantly. They have two data centers, each connected with two first tier suppliers, each providing two 2-Mbit lines. The datacenter are constantly exchanging backup data and transition logs.
Between their datacenters, bits are pushed through with 16 GB(yte) per second. Some more top server (Itanium 64) features:
- 32 CPUs and 64 cores
- 64 GB of memory (storing bets from the last two to three days)
- 64 network cards (requests are split per country, each country has a specific NW card)
Impressive facts. Thanks for the speech, I really enjoyed it.