Happy New Year and Welcome To The New Server Environment!

by GGBVanix on January 5, 2019 3:16:05 AM

It's a brand new year and it's been at least a week since I had to restart any services, so I guess it's safe to say that everything is now running stable. Esports Earnings (formerly "e-Sports Earnings") is now operating in a new server environment, with more powerful hardware at its disposal and a new software foundation on which new features will be built upon. This is the cumulation of over a year's worth of learning, designing, implementing, testing, and avoiding social media like the plague. I never thought constantly receiving messages for favors and sales offers would be so distracting, but isolating myself and cutting off all communications (or at least limiting it) proved to be essential. I apologize for having to give everyone the cold shoulder, but I felt that this required my full attention in order to address properly.

There's still a lot of changes that need to be made and a lot of features are still incomplete, but I had to perform the migration when I did because of a little something called "Feature Creep". When you have too many new features that need to be added, it feels like you never get to the end and the features you've added don't get added or used. In order to have the new hardware be utilized and the new changes go into effect as quickly as possible, I focused on the data entry forms. Many of you have already familiarized yourselves with the new features, but I'll go over the changes.

Server Rack


	This is a 22U server rack that I'm renting to house the servers. It is awesome. The former server environment was a single virtual machine with 3 cores and 6GB of RAM with everything running on it. Suffice to say, the slowdown during heavy traffic and the required processing time when new tournaments are added were becoming noticable. Limited RAM made in-memory caching impossible, and the shared disk I/O with other virtual machines made disk caching as slow as reading from the database. The new server environment separates the web service and the database. The database server has a whopping 32 GB of RAM, 8 1TB hard drives and a quad-core CPU, enough resources to implement the new features that require relatively large amounts of storage space, and has the I/O to support it. The web server has 8 GB of RAM, 2 1TB hard drives and a quad-core CPU, more than enough to handle the "Reddit Hug of Death". In addition, I've added a new system into the mix called "Memcached". This will cache database records in the memory of yet another server, and this will result in sigificantly faster load times for the site. There's still plenty of physical space available in the rack, which will allow me to add new features that require their own servers, or add more web servers to handle the increasing traffic. The possibilities are endless. Probably.

This is a 22U server rack that I'm renting to house the servers. It is awesome. The former server environment was a single virtual machine with 3 cores and 6GB of RAM with everything running on it. Suffice to say, the slowdown during heavy traffic and the required processing time when new tournaments are added were becoming noticable. Limited RAM made in-memory caching impossible, and the shared disk I/O with other virtual machines made disk caching as slow as reading from the database. The new server environment separates the web service and the database. The database server has a whopping 32 GB of RAM, 8 1TB hard drives and a quad-core CPU, enough resources to implement the new features that require relatively large amounts of storage space, and has the I/O to support it. The web server has 8 GB of RAM, 2 1TB hard drives and a quad-core CPU, more than enough to handle the "Reddit Hug of Death". In addition, I've added a new system into the mix called "Memcached". This will cache database records in the memory of yet another server, and this will result in sigificantly faster load times for the site. There's still plenty of physical space available in the rack, which will allow me to add new features that require their own servers, or add more web servers to handle the increasing traffic. The possibilities are endless. Probably.

Changes to the Database

The database was initially designed at a time when I knew very little about esports or what goes on in it. As such, quite a few things I added ended up being useless and bloated the database. Some relevant pieces of information were left out as well. And some things did not scale very well as the site grew. A few examples:

Some descriptions were truncated.
Some unicode characters were not stored properly.
Some players and teams were not credited and calculated properly.
Player selection via dropdown menu increases client memory and bandwidth usage.
Tournaments were not being associated with events in certain cases, and event dates might not update correctly.
New data could not be published while the site was recalculating existing data.

Many of these problems simply had "band-aid solutions" that only hid the problem to allow the site to continue working. "Squad Overrides" were unwieldy, but allowed certain players and teams to be accounted for in calculations. Player dropdowns consumed more memory and bandwidth as time went on to allow for the selection of more and more players. Out-of-memory errors happened every once in a while for these dropdown menus, and the quick fix to it was to simply increase the amount of memory the scripts were allowed to use; a terrible practice to be doing. 2018 was the year that I developed the permanent solutions to these problems.

Squads are now associated directly with every individual and team earning, and team and squad calculations will be based on that going forward. I also learned that "utf8" in MySQL wasn't true utf8. I should have been using "utf8mb4". Because reasons. I really don't know the whole backstory around utf8 and utf8mb4, but now unicode characters should be stored properly in the database.

Changes to the Development Area

The data entry forms have been completely overhauled to significantly reduce the number of requests sent to and from the server, and more information will be stored and rendered client-side via Javascript. In the old days, every form has simple "get" and "set" requests to the server, with the "set" requests automatically making a "get" request when successful. It was done this way because those forms were very simple to make and worked well for the time. As time went on and the website grew, multiple requests began to slow down both the server and the contributor's ability to modify data on the site. I'm sure some of you have noticed the longer response times. The new forms reduce the number of requests to the server; no more retrieving data for a specific team earning or squad from the server, as it will now be stored in your browser's memory when the page is loaded, and rendered locally (and instantly) when needed. When changes are submitted to the server, it will now respond with the updated data and render it in a single request.

One notable change to certain forms is the player selection box. This replaces the player dropdown menus (which becomes slower as more players are added) with a pop-up box that allows you to search the player you're looking for. This was a necessary change to increase the speed of your selection and save bandwidth by not requiring the server to send you several megabytes worth of player data to fill out a dropdown menu everytime you navigate to a tournament page. More ways to streamline this process will be added as they're discovered and implemented.

Tournament team earnings will now be much easier to add because player rosters will now automatically be detected and made available for selection. Contributors will no longer need to manually add each individual player to a roster, but that functionality will still exist to fine-tune the exact player rosters when necessary. Detection will be based on a tournament's end date and the players' join and leave dates, allowing for detection of current rosters at current events, as well as historial rosters at past events. This was one of the more tedious tasks I've identified and I was able to streamline this process. As long as the team data is kept up-to-date, this functionality will streamline the process of adding team earnings.

It has been a very meticulous and time-consuming process to rework these forms, but I hope that the amount of time saved in streamlining these processes will be well worth it to the contributors.

New Data to the Site

New data can now be added to the site, some of which is used to streamline the tournament data entry process, while others may lead to some interesting statistics for the esports community in the future. "Game Roles" are used to allow calculations to be further filtered by game-specific roles, such as Race, Character, Position, etc. So far, the roles are being used to calculate prize money by race in StarCraft II, StarCraft: Brood War, WarCraft III, and Command & Conquer 3. Without understanding "roles" in different games myself, I cannot implement them for every game. But the functionality to do this is ready to use for anyone involved in those communities who want to implement this feature.

Autocomplete Functionality

When searching for data, records can now be retrieved as you type. It has only been implemented in searching for players and teams for selection in forms, but will be further expanded for all data. Because this functionality requiures an obscene amount of processing power, it was near-impossible to implement it in the old server environment. Having a dedicated database server and memcached server means this functionality can run quickly and effectively.

Expanded Privileges

Long-time contributors now have the ability to publish their own changes, and will eventually have the ability to publish changes in the publishing queue. Regular, but fairly-recent contributors will be granted these privileges eventually as well.

There's still a lot of work that needs to be done for all the new features to come online and utilize the resources of the new server environment, but the hard part is now done. The site now has a fresh, stable foundation that new features can be built upon, and the site will now scale better as it continues to grow. Welcome to 2019!

GGBVanix