GRCRTools & Statistics for Grepolis

What makes up the GRCRT infrastructure and why did we make the decisions we did?

We went through different infrastructure options and drew conclusions. Below we briefly describe what we tried up (and with what result).

Which architecture?

VPS - of the many tested, unfortunately none worked. Mainly due to resource management by the hypervisor and also the restrictions (often hidden) imposed by the hosting.

Dedicated server - seems to be a good solution, but here the challenge is IOPS (disk access time). On harddrives, IOWAIT could reach 30% of the time. SSD/NVMe drives, on the other hand, have another disadvantage: the minimum block size is 4KB, and with a lot of small files ~300B, yes, the space is there, but no inodes :/

We wenth with both NVMe and harddrives, in pairs in RAID1. The NVMe drives hold what needs to be accessed quickly (database, user records, images) and the harddrives hold what might run slower (php and perl scripts, json files for images etc).

Which web server?

Apache? - works well with perl (more on that later) and php, but is too slow.

Nginx? - can't handle perl properly.

OpenLiteSpeed? - crashed under heavy load.

What is left? NGINX + Apache - it's almost perfect. Almost, because users on the other side of the globe are queuing for images etc. CDN solves this problem and, in the process, relieves the load on the server.

Loading data?

Every hour the server downloads all the data from all active InnoGames servers and loads it into the database. Our proprietary bash+perl mechanism, in addition to downloading and loading data, detects if there is nothing to download from a server (because it has been closed) and also whether a new server has been opened. The data is updated in such a way that unavailability to users is less than 1s, while the actual time it takes to download and load it into the database (currently 1591 files for all active worlds) is: ~15m

We tried various database engines, but MySQL proved to be the fastest (table partitioning, text file loading speed)

Alternative sources of funding

We tried to get any support from InnoGames, but they were not interested. Instead, they were happy to implement our solutions in the game (for example, the city resercation module or the hero coin count display) 😉

There are no ads for two reasons. Firstly, there were advertisements for adults which we didn't want, but also had no control over. Secondly, InnoGames banned the display of in-game ads. We decided that the site didn't need to be one big billboard and bet on donations - which kind of worked out for the first 2 years.

In summary, the hardware part of our project

main server

CPU: Intel Xeon E3-1230v6 – 4c/8t – 3.5 GHz/3.9 GHz
RAM: 32 GB ECC 2133 MHz
drives: 2×450 GB SSD NVMe (Soft RAID), 2×2 TB HDD SATA (Soft RAID) – just to store ~3,000,000 files.

Backup server (on another continent)

as it is not a subject to as much load as the main server, a fraction of its parameters will suffice. It can't be an Atom though, because it's too slow. So any Intel Xeon with sufficient disk space. The amount of RAM does not matter.

The current parameters of the backup server, are:
CPU: Intel Xeon E3-1225v2 - 4c/4t - 3.2 GHz/3.6 GHz
RAM: 16 GB 1333 MHz
drives: 3×2 TB HDD SATA Soft RAID (RAID5)

Have questions?

Reach us out on our Discord (link in the footer of this page)