EPTWe're Getting A New Server!
Ever since I started volunteering for Team Liquid, I've been responsible for the server handling all of our websites. Yes, that's singular - TL, Liquipedia, Liquid Dota, Liquid Hearth, Liquid Legends and our team site all run from one server! When TL was created back in 2001, the site was running on shared web hosting combined with some free ISP web space. Things have changed a lot since then - we've made numerous upgrades over the years, some notable ones being for TSL and then to handle the traffic from SC2.Our last major upgrade was in 2010 when I started working for TL full time right as the SC2 traffic was starting to ramp up. We purchased a dedicated server at Voxel (later bought by Internap) for a great price at the time through some contacts. Over the years we made some minor upgrades, doubling the RAM and adding a larger SSD, but seven years later it's time for something new (and I'm also afraid of the HDDs dying any day now). Here's our new server at OVH which will hopefully be entering production some time this week.
| Old Server | New Server | |
|---|---|---|
| CPU | 2x Xeon L5630 @ 2.13 GHz 8 Cores, turbo boost 2.4 GHz (4 cores) | 2x Xeon E5-2690v3 @ 2.60 GHz 24 Cores, turbo boost 3.10 GHz (24 cores) - 3.50 GHz (4 cores) |
| RAM | 32 GB DDR3 | 256 GB DDR4 |
| File Storage | 2x 146 GB Seagate 10k RPM SAS HDD MegaRAID HW RAID-1, ext3 | 2x 800 GB Intel SSD DC S3500 ZFS mirror |
| DB Storage | 1x 240 GB Intel SSD DC S3500 ext4 | 2x 800 GB Intel SSD DC S3500 ZFS mirror |
| Network | 1gbps with data limit No DDoS protection | 500mbps w/3gbps burst, no data limit DDoS protected |
| Backup | Daily, proprietary (R1Soft), 200 GB space | Hourly, off-site ZFS snapshots, 2 TB space |
| Linux | Debian, 3.16.0-4-amd64 | Debian, 4.9.0-0.bpo.2-amd64 |
| Location | NYC, United States | Montreal, Canada |
From the specs alone, this is a significant upgrade over our current hardware. With the 4th generation Xeon CPUs running at higher clock speeds, webpage generation should be much faster overall, especially on Liquipedia where non-cached pages are CPU bound. The large amount of RAM keeps our entire database resident in memory and allows us to bump up the size of our Varnish storage for cached wiki pages. The increased number of cores enable more simultaneous pages to be rendered; the current server quickly maxes out during big events due to heavy editing traffic on Liquipedia.
Moving away from a proprietary backup system (R1Soft requires binary kernel modules) allows full flexibility with the kernel and I've chosen to use ZFS on Linux along with the latest Debian backported kernel. ZFS is an enterprise copy-on-write file system with excellent data integrity features, compression, snapshots and more. The new kernel also makes technology like Google's BBR TCP congestion control available, which greatly improves data transfer speed over higher latency connections.
The server location has moved from Internap in New York City, USA to OVH in Montreal, Canada. This should have a minimal effect on latency and it allows us to have geo-distributed backups - the current server only backs up to the same datacenter in NYC whereas the new server backs up to a different continent entirely (to a server in France). While OVH do have polarizing reviews, I've been running my personal sites there for a while without any issues and feel confident enough to use them for TL. I have a feeling that a lot of the negative reviews come from people who don't understand that they are a fully unmanaged provider.
I also took the opportunity to move all our background processing tasks such as deferred thread updates, live stream info, spam detection, etc into systemd units, so they are properly managed and restarted if any issues occur. This should finally put things like this issue to rest.
