For today’s interview, we have Daniel Porteous, a Production Engineer on the IaaS Resources team at Meta. The IaaS Resources team is responsible for building systems that move machines between ownership domains without compromising on races and errors that lead to accounting issues. While working on this team, Daniel has been using Rust as one of the main languages for development. Most of his work these days centers around the Twine Experimentation Framework (TWEF), which enables users to acquire hardware, apply host level changes, start workloads, run custom automation, and ultimately clean up changes safely, all with a single request. Let’s hear from Daniel about how his experience with Rust has been and learn more about his work.
In my current team, we use Rust on a day-to-day basis for every service and almost every tool that we own. While we maintain some existing tooling in Python, and support a team that has a massive, complex C++ codebase, we build all new services and tooling in Rust.
Twine Experimentation Framework (or TWEF) workflows can take a long time to run due to downstream dependencies, often up to an hour. Previous experimentation systems have been written in Python and C++ and were prone to bugs that would cause requests to fail deep into the workflow, which led to user frustration. Rust eliminates entire classes of bugs and catches many simple mistakes at compile time, like comparing optional values to non optional values, allowing us to focus on other areas.
TWEF has gone through many iteration cycles, through which the implementation for pieces such as acquiring hardware and starting workloads has gone through many iterations. With Rust's trait system, we are able to easily define interfaces to each of these subsystems and swap out the implementation without changing the rest of the service. This allowed us to rapidly develop without sacrificing our ability to improve over time.
TWEF is one of many Rust projects I have worked on at Meta. What is interesting about TWEF is it uses a workflow engine that currently does not support Rust directly. That did not discourage us from using Rust. For the worker part of this workflow, we built a thin Python layer that calls into a Rust library that does the actual work. The thrift service that sits atop this workflow is also written entirely in Rust. This was ultimately completely worth the effort and has helped us avoid many errors that could arise in other languages due to the complexity of the requests TWEF processes.
I believe one of the key factors that allows Rust to grow at Meta is how easy it is to use alongside other languages, such as the Python -> Rust layer I described above. Operating with C++ is even easier due to CXX, which can produce safe bindings to existing C++ libraries.
Beyond the language itself, I feel the Rust community within Meta is extremely collaborative and friendly. We all strongly believe in the language and work together to improve the user experience to try and grow its presence within the company. I feel with this group of Rustaceans we will one day make Rust the first-choice language at Meta.
I feel like Rust is taking off across the entire industry—big tech companies, game development, embedded open source side projects, you name it. I feel pretty lucky that I was able to get into Rust so early because I'm confident that in the years to come it will become the main language for many, many use cases.
For me it is the documentation and community. While I of course love the Rust compiler and its expressive feature set, I'm consistently impressed by the quality of the Rust documentation, particularly how the docs are full of great examples of how to use whichever feature you're looking for help with. This extends to both the regular docs as well as the various Rust books out there, which are also incredible. The community itself is very welcoming and is always willing to help. I always get excellent answers on Stack Overflow and in the Rust Discord.
The Rust Book is by far the best way to learn about Rust. Rust and popular Rust crates generally have excellent documentation, with great crate / class level explanations and examples provided for common use cases. This emphasis around leveraging examples for learning extends into code too; cargo provides first class support for running examples and most popular crates leverage this functionality. The Rust Discord is also very welcoming to new users.
We would like to thank Daniel for taking the time to do this interview. It was very interesting to learn how Rust is being used as a primary language for TWEF and other systems, and how we are learning from all the things we build here and contributing back to the Rust community. We hope you found this interview useful and it gave you some insight into how and where Rust is being used at Meta. Follow Daniel on Twitter at @banool1.
Check out our previous blogs from the Meet the Rustaceans series:
Meet the Rustaceans: Chris Konstad
Meet the Rustaceans: Pedro Rittner
Meet the Rustaceans: Neil Mitchell
Meet the Rustaceans: Daniel Xu
Meet the Rustaceans: Eric Kuecks
Meet the Rustaceans: Gus Wynn
Meet the Rustaceans: Brendon Daugherty
Meet the Rustaceans: Hasnain Lakhani
Meet the Rustaceans: Digant Kasundra
Rust has consistently been ranked as the “most loved” language for the last 5 years and we at Meta believe that Rust is an outstanding language that shines in critical issues such as memory safety, performance, and reliability, and is being used widely over a large range of projects here. We joined the Rust Foundation to help contribute towards the improvement and growth of Rust, which not only strengthens our commitment towards the Rust language but also towards a sustainable development of open source technologies and developer communities across the world.
This blog is a part of our Meet the Rustaceans series, where we invite the engineers and developers who use Rust on a regular basis to share their experiences and tell us about the amazing products that they are building using Rust here at Meta. Look out for more interview blogs where we meet with many more engineers and hear their thoughts on this topic.
Interested in working in Production Engineering at Meta? Check out our job postings on our Production Engineering career page here.