Announcing Ringmark, a Mobile Browser Test Suite

Introducing Ringmark

As part of our work on the mobile web over the past year, we've often wished that there had been a test suite for mobile browsers that is comprehensive, fair, and that tests the feature sets that developers really need.

We decided to solve this problem and solved our own need. Today, we're proud to announce Ringmark, a new mobile browser test suite which does just that.

Ringmark, developed together with Bocoup, helps you understand which mobile browsers support the functionality your app needs.

You can access it on your mobile device at http://rng.io, and in the near future, we will be open sourcing and donating these tests to the W3C.

Anatomy of Ringmark

The core concept behind Ringmark is based around the question of "what do developers need?"

Specifically, Ringmark is comprised of tests of core functionality that web developers need in order to build their apps. For example, orientation lock, which is important for games. Another example is camera functionality for social apps.

While helping mobile browser vendors test key features, it can also help them prioritize features by providing a clear road map for what is important for developers.

Ring Methodology

You can think of the rings as straightforward software versioning. Ring 0 represents the base functionality that most mobile phone have today. Ring 1 represents what functionality is needed to unlock the most common apps that developers want to build; specifically, 2D games, music and video apps, and camera apps.

From there on, each subsequent ring represents a slice of features that will unlock the next generation of mobile web apps, based on developer necessity. For example, we expect Ring 2 to include upcoming technology like WebRTC and WebGL.

When you run rng.io, each ring has a comprehensive list of tests in order to test the features in each ring.

There's more to come

Like the web, Ringmark is a work in progress and we appreciate any feedback you have. You can start by leaving a comment below.

We encourage you to open rng.io on your mobile device, contribute to Ringmark once it's open source, and like the Facebook Page for updates. Also, join the conversation in the HTML5 Facebook group.

With that, I'd like to introduce Boaz Sender of Bocoup, with whom we worked on developing Ringmark, to explain the technical details of the test suite.

Bringing Practicality to Test Suites

As Matt said above, one of Ringmark's primary goals is to prove that web features work for mobile web developers, so we knew we had to write deep tests, ones that went above and beyond answering "is window.awesomeNewAPI something besides undefined?" As of today, our tests run the gamut from reliable inferences to thorough checks of whether an API is truly usable and produces the expected, specified output.

Today, the Ringmark suite already contains over 400 tests. Over time, we'll continue to add tests to measure performance under heavy use and improved functional tests that prove features really work as they should.

Figuring out what to test is one thing, figuring out how to test it is entirely another. To that end, the project owes an incredible debt of gratitude to the pioneering open-source projects in this space whose techniques and research have proven invaluable. So if you've ever contributed to Modernizr, caniuse, Are We Playing Yet, or the W3C Tests, thank you!

Ringing Everybody In

Throughout this process, the W3C has been involved to figure out how our progress on Ringmark could be used to improve the standards process. A keen focus has been making sure the tests are compatible with the existing W3C test harness that runs the existing approved W3C HTML Tests that OEMs already use. Maintaining that compatibility was crucial, as we intend to donate these tests to the W3C in the future. Special care has been taken to differentiate support for vendor-prefixed implementations from actual support for features as specified by the W3C. And of course, we'll be writing more tests as specifications become stable in the future.

Ringmark's tests have been constructed so that the suite structure will be familiar to anyone who's already contributed to any project with a large, client-side unit test suite in place, with tests grouped into modules by W3C specification, and written using QUnit's assertions, with a simple fixture if necessary. Tests for individual modules can be run atomically, and the actual Ringmark site is just a specially built flavor of output for the entire test suite.

In the near future, we'll be working further with Facebook to open-source Ringmark and publish it on GitHub. We believe that this will open a productive avenue for developers to take concrete steps to get involved in pushing the web forward.

If You Like It...

We're all looking forward to inviting you to engage in this research and testing endeavor with us. In the meantime, please run Ringmark on as many devices as you can, share your feedback, and stay tuned via the Facebook Ringmark Page.