Open Source: 2021 Year in Review
This article was written in collaboration with Dmitry Vinnik, an Open Source Developer Advocate at Meta.
- Meta Open Source by the Numbers
- Foundations and Partnerships
- Community
- Meta Open Source Portfolio at a Glance
This past year has been full of challenges and opportunities. We pursued our mission of empowering diverse communities through open source technology with conviction. We launched many exciting projects and received contributions from developers worldwide, growing the open source ecosystem. In this post, we would like to appreciate and highlight some of the achievements and contributions from 2021 that could not have been possible without the support of the open source community.
We will summarize key metrics and detail our foundations and partnerships' focus over the last year. We will review our community engagement through the Meta Open Source social media channels like Twitter, YouTube, the Diff podcast and our blog. And lastly, we will categorize our open source portfolio by technology domain such as Developer Tools and Programming Languages, Data and Infrastructure, Mobile, Web and AI/ML so that we can share a snapshot view of how our portfolio is evolving.
Meta Open Source by the Numbers
At the end of 2021, we had 837 active public projects. After discounting inactive projects that were archived, or projects that were gifted to foundations, that represents 231 new projects launched this year.
Over the past year, we also saw 165,163 commits to our public projects. This number is nearly 20% larger than in 2020. Of those commits, 23,938 were made by community members not employed at Meta. Meta employees made the remaining 141,225.
Our projects on GitHub accumulated an additional 133,854 stars, bringing the total to a staggering over 1.4 million. The community shows accelerating interest and excitement for Meta Open Source projects.
Meta Open Source projects are made possible because of contributions by developers like you. Pull requests, conference talks, feedback on documentation, Facebook posts and anything and everything in between—make our projects better and strengthen the connection in our communities. Thank you for making 2021 another stellar year for open source.
Foundations and Partnerships
We believe that open source allows developers to work together uniquely and ambitiously, allowing us to achieve common goals. Our mission is to empower diverse communities through open source technology, and we always look for opportunities to be involved in the appropriate foundations and partnerships that advance this mission.
In early 2021, we announced our support of the Rust Foundation, at its highest member tier, to sustain and grow the Rust open source ecosystem and community. As a part of the foundation, we've worked with the other members to announce cloud compute resources for Rust language development. Our support allows the project to evolve while reducing the time spent building and testing language changes and updates.
We also lead funded Open Web Docs. We joined their Governing and Steering Committees to shape the project and deliver social and economic benefits through well-maintained web documentation. Our sponsorship contributed to the hiring of full-time Open Web Docs technical writing staff, and to the future of web platform documentation, adding 131 new pages on WebXR.
In addition to these new partnerships, we continued to collaborate and participate with existing foundations from last year. As a member of the Presto Foundation - a project we gifted to the Linux Foundation - we helped organize PrestoCon Day, which was a great success in bringing the Presto community together. The foundation shared recordings of the talks on their YouTube channel following the event.
As an active member of the Linux Foundation, our involvement goes well beyond Presto. In 2021, Meta was the largest contributor to the kernel directory, with 17.6% of all changes by developers at Meta. We also moved one of our open source projects called Ent, an entity framework for Go, under the Linux Foundation to aid in its development and foster the community of developers and companies using it.
In early 2021, we also worked with the Linux Foundation to establish an industry collaboration for Magma, a platform for building scalable carrier-grade networks, to help accelerate the deployment of wireless networks and deliver innovative, next-generation services and launched Magma Core Foundation with over 20+ members. Later in the year, we partnered with several companies to launch the eBPF Foundation to help to redefine networking, security, tracing and observability.
Community
Community is what defines open source. We want each of our projects to grow their communities while making it a welcoming space for everyone. We shared our developer programs in more ways than ever before.
In 2021, we continued our partnership with Major League Hacking on the MLH Fellowship program, designed to host, mentor and support developers hacking on open source projects. We believe that investing in open source is a way to empower developers and make the community more diverse. To spotlight and share the success stories of the MLH fellows, we published a Contributor Story blog series. We will continue sharing other posts with similar content on our open source blog, and we hope this will encourage people to contribute to open source and become a part of our growing community.
Although most conferences and events in 2021 took place virtually, we didn’t want to miss the opportunity to build a dialog with our community. We delivered several talks that encouraged everyone to use and contribute to open source and work together to make it easier to build great products for everyone. We presented sessions like The Tale of Five Projects: A Reflection on the Motivations Behind Open Source at Facebook, Hands-on React Native: From Zero to Hero and many more.
As we wrapped up the year, we re-launched our podcast, the Diff, adding a video component to reach an even larger audience. In the latest episode, we interviewed Paul O'Shannessy, Meta Open Source Program Engineering Manager and talked about starting and managing open source projects at scale.
We also participated in Q&A sessions with the Major League Hacking (MLH) fellows and the F8 Refresh Hackathon. We shared best practices and discussed other nuances specific to getting started with open source. We also continued our ELI5 (Explain Like I’m 5) series to give the community access to a comprehensive range of topics in a way that is easy to understand and use.
To ensure that we continue modernizing how we inform and share projects with our community, we launched a new website built with Meta Open Source projects like Docusaurus, React and Jest in June 2021.
We hope this will guide current and future community members on their exciting journey.
Meta Open Source Portfolio at a Glance
After covering some metrics around Meta Open Source, our external collaborations and community engagement, we want to highlight the components of our open source portfolio. This section will look at our open source contributions in different areas like Developer Tools, Programming Languages, Data/Infrastructure, AR/VR, Mobile/Web and AI/ML.
Developer Tools
Here at Meta, we want to empower engineers to be more efficient and productive. One way to do that is through developer productivity tools, which we are happy to share with developers publicly.
Recently, we described the future of our flagship build product, Buck, an open source system used to build in 15+ languages targeting different platforms ranging from mobile devices to web services to VR headsets. The next version of Buck is being re-architected from the ground up, using Rust throughout its technology stack to deliver very high performance and embrace extensibility. Various components of the next version of Buck have been open-sourced, including Rust-Starlark and Gazebo, and we expect more to be open-sourced in 2022.
We released the next version of the open source documentation infrastructure tool, Docusaurus 2, which went into official beta in 2021 and is nearing full release.
The developer tools space is broad and far-reaching. In 2021, we released a project, Winterfell, which delivers to all developers proofs of cryptographic computational correctness that would otherwise require specialized knowledge and skill to produce. Soon after that, we open-sourced a tool called Below that “travels” through time, helping teams to view and record historical Linux system data.
We believe that the best way to battle test an open source project is to use it in production. This reason is why our static analysis tool, Infer, was used to help with rearchitecting Facebook Lite by catching race conditions and other performance glitches at compile time.
Programming Languages
There is no one default programming language at Meta. From third-party languages such as Swift and Java, to homegrown developed languages Hack and Flow, our engineers use the best language for the job. For example, we published Hsthrift, bringing the benefits of Thrift's simple, language-agnostic protocol for communicating with structured data, to Haskell. We also published an entire series on async stack traces in the C++ developed library Folly, and helped to make Python a more efficient and high-performance language with Cinder, Instagram's internal performance-oriented production version of CPython 3.8.
In 2021, the Open Source team focused much of their efforts on one particular language: Rust. Rust is fast becoming a favorite at Meta and in the developer community at large. We even formed a dedicated Rust team to grow Rust development inside Meta, encourage open source contributions to Rust and Rust-based projects, and improve engagement with the Rust community. We launched two series showcasing the use of Rust within Meta: Meet the Rustaceans shares stories about our developers who use Rust in their day-to-day work, and Rust Nibbles, which dives into a specific topic or project in “bite” size pieces. An exploration of Gazebo was the first in the Rust Nibbles series.
Data and Infrastructure
With such a breadth of services and a massive user base, scale is one of the biggest challenges we address. And true to our mission, we love to share our innovations in data processing and infrastructure with the open source community.
In early September 2021, we open-sourced a caching engine for web-scale services called CacheLib. This project that we created in close collaboration with partners like Twitter, Carnegie Mellon University, Intel and many others, aims to help engineering teams build and scale high-performance services across numerous use cases. We also published a new benchmark called CacheBench to let teams objectively evaluate caching performance on diverse production workloads.
At Meta, we are trying to solve some of the same problems that many other contemporary organizations are working on, including those related to time and data. We built and made public an Open Compute Time Appliance that plays a significant role in modern timing infrastructure, making it more secure and affordable. In addition to this hardware project, we open-sourced a new library called Kats for analyzing time-series data. This project is a first-of-its-kind, comprehensive Python library that lets its users explore time series with both classical and advanced techniques. This type of data is vital in anomaly detection, feature extraction and analysis, which are fundamental parts of many industries like e-commerce, medicine and supply chain management.
AR/VR
We expanded our community presence regarding AR/VR efforts at Meta. As a part of this initiative, we recently launched Shared Spaces, the Oculus multiplayer showcase samples, demonstrating basic multiplayer functionality in Unreal and Unity. These include Oculus Social APIs, Oculus Platform authentication, Photon Realtime, Photon Voice with Oculus Spatializer and much more. This Connect session from 2021 explains how to use these samples to build and grow multiplayer apps for Quest.
At the GlobalXR Conference 2021, Developer Advocate Navyata Bawa spoke on behalf of Meta on "Fostering Inclusivity in VR" for the event's closing keynote, covering various topics, including developing open and inclusive communities, Universal Design, Oculus Virtual Reality Checks and more.
Mobile and Web
Our family of apps like Instagram, Facebook App, Oculus and more, have led to advances in front-end technologies that we are excited to share with the open source community.
We open-sourced Mariana Trench, a tool that lets users analyze Android and Java app security in depth. Considering the millions of lines of code that our developers at Meta are working on, we need to take considerable advantage of automation to scale our security testing. We build sophisticated systems that allow our security engineers to automatically review code and detect potential issues, shifting away from purely manual code reviews. We are excited to see that the open source community found Mariana Trench valuable, which helped some of our engineers win the 2021 IEEE Computer Society Cybersecurity Award.
React Native has continued raising the bar for mobile development, both at Meta and elsewhere in the industry. In late August, we published React Native’s Many Platform Vision to bring the framework to desktop and virtual reality platforms. The team shared its plans to expand to new platforms with apps like Messenger for Desktop, now available on Windows and macOS. React Native also announced a closer partnership with Reality Labs, to help bring the framework over to VR.
Continuing to share our updates in the React ecosystem, the community has been vital in developing and adopting a new version of React. Earlier in 2021, we created a public workgroup to help develop React 18 with features like Concurrency support, automatic batching for fewer renders, Server-Side Rendering for Suspense and many more. We also engaged with the community further and had a blast sharing some of the fantastic work that the community has done at ReactConf 2021. We launched a new React Docs website, designed around the learning experience, to help folks learn React more efficiently.
We also launched a new React Labs video series to share ongoing research and development done by the React team. As part of this series, we did several Q/A sessions about the React Server Components and updates for React 18.
To further evolve the web tooling space, we made significant improvements to Relay, a JavaScript framework for fetching and managing GraphQL data in React applications. In March 2021, we introduced Relay Hooks, a new, more developer-friendly Relay API built using React Hooks. This latest iteration of Relay aims to improve developer experience significantly, have better type-safety with greater coverage and avoid error-prone tasks like re-fetch queries.
AI/ML
Open datasets and benchmarks have been key drivers of recent advances across AI. To promote open and collaborative research for more languages worldwide, Meta AI released datasets for multilingual speech research, two datasets for semantic parsing in conversational AI and the most extensive multilingual speech corpus for AI translation.
Last year, Meta AI researchers went beyond text by training NLP models directly on raw, unannotated audio signals, making AI more inclusive and able to model a wider variety of languages. Meta AI also has open-sourced Blender Bot 2.0 that can remember conversations over weeks and months, and generate contextual internet search queries while responding to people’s questions and comments.
We also shared new research on how we teach AI to perceive the world through our eyes (and perspectives) instead of from the sidelines, to help robots understand touch, and a simulator to train home assistant robots.
It's truly exciting that all of this cutting-edge research is powered by PyTorch, a popular open source machine learning library. As the biggest contributor to PyTorch, Meta continues to evolve the project with almost 400 contributors who pushed out two new PyTorch releases (1.8, 1.9 and 1.10) last year, improving support for distributed training, scientific computing, edge computing and inference performance. To engage with the open source audience, the PyTorch team hosted Ecosystem Day and Developer Day, where community members met online, engaging in technical talks at the cutting edge of Machine Learning, deep dives into ML projects and discussing the latest and greatest of PyTorch. The PyTorch Developer Advocate team led by Suraj Subramanian also launched PyTorch Community Voices, a weekly live chat show on YouTube. This new series highlighted external contributors who presented their PyTorch-based projects and shared their use of AI to tackle diverse problems ranging from healthcare imaging, deep learning for time series and libraries that improve developer productivity.
We at Meta Open Source believe that open source accelerates the pace of innovation in the world. By sharing our research, code and designs with others, we're moving the industry forward while helping other companies and individuals scale more quickly and build great products.
We are very thankful to our worldwide community, who came together to contribute their efforts and time creating and growing open source technologies that anyone can freely use and contribute to. We look forward to working with everyone in the years to come.
To learn more about Meta Open Source, visit our website, subscribe to our YouTube channel, or follow us on Twitter and Facebook.
Meta Open Source: Nhìn lại năm 2022
