In this article, web engineers Antoni Choudhuri and Brandon Chothia discuss ROBLOX’s commitment to quality via automated web testing, and what the future holds for our testing infrastructure.
ROBLOX is a growing company – just look at our Jobs page to see the variety of positions we’re looking to fill – and our expansion is enabling us to develop and release features faster than ever before. While we’re producing a greater quantity of features, we’re just as importantly placing a greater emphasis on the quality of everything we do. One of the ways that has manifested itself is the expansion and improvement of our testing infrastructure for Roblox.com.
When you have a website with bundles of deep-rooted, interconnected features and pages, you encounter a challenge: every change you make has the potential to affect something, somewhere down the line. A ROBLOX developer could, for instance, make a change to the Money page that affects something as seemingly disparate as the Games page. Being that we’re constantly developing Roblox.com and it’s a rather large website, we rely on automated web testing to ensure a high-quality user experience.
Automated web testing involves engineering tests to replicate the behavior of a website user. At ROBLOX, we use Selenium, a popular open-source platform, to automate the testing of many user activities: registration, logging in, viewing user profiles, making friends, posting to the forums, buying items, making trades, and so on. We have engineered hundreds of tests for Roblox.com – over 20 each for the Trading System and Add Gear to Game features, to put it in perspective – and we run the full battery in a test environment multiple times per day.
Writing an automated test isn’t as simple as “recording” yourself taking specific actions. Instead, test cases are written the same way engineers develop Roblox.com – by writing code (specifically in C#). This allows us to engineer complex interactions – things like proposing and completing virtual item trades with added currency – and replicate the behaviors of actual ROBLOX users. We also use the Selenium API to automate interactions with the browser. For example, a simple action like clicking a button requires us to find the element on the page and invoke a click method on that element to make the mouse automatically interact with it.
Every test runs at almost inhuman speed, and we can watch video recordings of them after they’ve happened. Here’s an example of a simple test that registers a new user, clicks through every link on the header, and verifies that each link leads to the intended page without errors.
If anything in the testing process fails or changes – for example, the name of a button changes from “Join” to “Submit” – the test will fail, and we quickly know where to find the error. This keeps known failures in functionality from ever reaching the live website.
Sample automated web testing results; click to enlarge
Automated web testing is a significant part of ROBLOX’s greater web-QA infrastructure, which is three layers deep. The first layer involves the web developer manually testing new code on a test environment known as Site Test 3. It then undergoes automated testing on Site Test 2, before making its way to Site Test 1, where users can actually interact with the update and provide feedback via dedicated testing forums on ROBLOX. You can read more about user testing in this previous blog post.
Using Selenium, we run our suite of tests across the most popular browsers and operating systems among ROBLOX users. You can see the dispersal of browsers among ROBLOX users in the below chart.
If one theme of automated web testing is quality, the other is efficiency. So, while we have hundreds of tests, each one taking an average of one minute to complete, we don’t want to have to wait hours to get our results. Instead of waiting, we run tests in parallel; this cuts the testing time by a significant fraction and allows us to return complete results quickly. This is one of the key areas we’d like to ratchet up in the future and, as our team expands, the number of tests we can run in parallel will be a key. Ideally, we’d be running enough tests in parallel to always have results within an hour. That’s the gold standard.
This leads directly into another goal for the future: to have automated testing coverage across all of Roblox.com. We’ve already engineered hundreds of tests, but there are pages that we have not yet covered. We’d also like to support even more browsers and operating systems, and run automated tests on ROBLOX Mobile, which uses modified versions of desktop-view web pages as a launch pad for games, catalog purchases, your inbox and more.
There are millions of ROBLOX users visiting our site and mobile app each month, and they’re not making quick appearances; in 2012, users logged as many as 13.3 billion pageviews in a single month. That speaks volumes about the nature and interconnectedness of Roblox.com, not to mention the importance that all the wires be connected just right. A suite of automated tests running in parallel is unquestionably one of the keys in our relentless pursuit of quality across all those billions of pageviews.