Meet the Rustaceans: Daniel Xu

This article was written in collaboration with Daniel Xu, a Production Engineer at Facebook.

For today’s interview, we have Daniel Xu who is a Production Engineer on the Linux userspace team at Facebook. The Linux userspace team works on a wide variety of systems tools, libraries, and services, all of which deal directly with the Linux kernel. While working on this team, Daniel has been using Rust as one of the main languages for development. Most of his work these days centers around observability, particularly dynamic tracers and debuggers. Let’s hear from him about how his experience with Rust has been and learn more about his work.

In what capacity have you been using Rust?

I've used Rust to write services, libraries and standalone tools over the last year and a half to solve low level problems such as making system calls, parsing binary data, making Foreign function interface (FFI) calls to C libraries, etc. These projects have high correctness requirements and any bugs would cause issues that would be very hard to debug or require specialized knowledge for the end user.

Why did you/your team at Facebook choose to use Rust over other languages?

I had heard all the fanfare surrounding Rust and was naturally hesitant - choosing a programming language for a non-trivial task is an important design choice. A wrong choice could lead to frustration, technical debt, or worse. Luckily for me, my coworker, who is a long-time systems expert and C++ programmer, was also a fan of Rust and encouraged me to try it out. On their advice, I took a leap of faith and started writing Below, an interactive tool to view and record historical system data, along with a few other enthusiastic colleagues.

On paper, Rust seemed like a natural choice. Below had to parse a lot of textual data, it had high uptime requirements, and it had to be efficient. These factors play into Rust's strengths. Rust's emphasis on memory safety by default and correctness-if-it-compiles allows us to move faster and be more confident that we're writing correct and safe code. The deal is further sweetened by the fact that Rust compiles to native code and has a relatively lightweight runtime, when compared to Python/Go/Java - every bit of efficiency matters when Below is running on millions of hosts.

What are some of the projects that you’ve worked on at Facebook that use Rust?

Below is one of the major projects I've worked on in Rust. Below is a system monitor: think of it like an interactive process viewer such as top or htop, except that it records historical data to disk and is also cgroup aware. Engineers across the company rely on Below to debug system issues on any of the millions of the bare-metal hosts the company manages.

Another project I've written in Rust is libbpf-rs, which provides safe and idiomatic bindings to libbpf (a C library) that allow Rust developers to work with extended Berkeley Packet Filter (eBPF). This project was developed in the open and is currently used by a number of external developers. To learn more about libbpf, visit my blog, where I've written more extensively on this topic.

Two other fairly niche projects I've worked on are btrfs-fuzz and btrd. Btrfs-fuzz is an unsupervised, coverage-guided fuzzer for btrfs, a next-generation Linux filesystem, while brtd is a filesystem debugger, which allows file system developers to directly view the on-disk data without looking at literal ones and zeros.

In all the above projects, I've been incredibly surprised with the care the Rust developers have put into developing the language. I’ve used Rust as an application developer as well as a library developer. As an application developer, I've loved how you can write relatively well-performing code by writing idiomatic Rust; that idiomatic Rust can look and feel fairly similar to a scripting language; and the fact that if your code compiles, you're in pretty good shape, means that systems programmers really can have nice things too.

As a library developer, I've loved how Rust provides powerful facilities to build abstractions that resist misuse. Ownership and borrowing are concepts that systems programmers are intimately familiar with. To have them baked into the language and enforced at compile time represents a huge leap forward in writing correct code.

How do you feel about Rust's growth trajectory at Facebook?

I feel pretty good about Rust's trajectory. Facebook owns a lot of physical hosts and our engineers are exposed to all sorts of problems related to systems. Rust has been fairly well received internally and is getting more attention all the time. I believe that there is and will remain enough internal expertise for other teams to confidently choose Rust for their next project.

How do you think Rust is growing as a language in 2021?

I think Rust is nearing or past the peak of its hype cycle. Personally, I think that's a good thing. There's that old adage: "There are only two kinds of languages: the ones people complain about and the ones nobody uses." I don't take complaining to necessarily be a good sign, but I do think it means people are starting to turn a more critical eye towards Rust. I believe that's a good thing: nothing is perfect and things can only be improved if there is feedback.

Some people who have used Rust have come to really like it. Why do you think that is and what is your favorite feature about Rust?

It depends a lot on what a person is doing and in which space they're working. For me, I love the compile time guarantees. I come from a C++ background so I've been used to using (and loving) ASAN (AddressSanitizer)/TSAN (ThreadSanitizer) and all the other sanitizers. What really sold Rust to me was when someone told me that Rust doesn't (yet) have any sanitizers because it's all enforced at compile time by the compiler.

Where can people learn more about Rust and how can they start contributing to it?

I think the best way to learn any programming language is to start writing code. Write some non-trivial code and just google search all your questions.

As for contributing: you don't necessarily have to have any experience working on programming languages. Almost all open source projects are in great need of bug reports and testers. The easiest way to get started is to try out libraries, nightly builds, tools and report back any bugs you see.

We would like to thank Daniel, for taking the time to do this interview. It was very interesting to learn how Rust is being used as a primary language to build a wide variety of systems, tools, libraries, and services and how we are learning from all the things we build here and contributing back to the Rust community. We hope you found this interview useful and it gave you some insight into how and where Rust is being used at Facebook. Look out for more interview blogs where we meet with many more engineers and hear their thoughts on this topic.

Check out our previous blogs from the Meet the Rustaceans series:
Meet the Rustaceans: Chris Konstad
Meet the Rustaceans: Pedro Rittner
Meet the Rustaceans: Neil Mitchell

About the Meet the Rustaceans series

Rust has consistently been ranked as the “most loved” language for the last 5 years and we at Facebook believe that Rust is an outstanding language that shines in critical issues such as memory safety, performance and reliability and is being used widely over a large range of projects here. We joined the Rust Foundation to help contribute towards the improvement and growth of Rust, which not only strengthens our commitment towards the Rust language but also towards a sustainable development of open source technologies and developer communities across the world.

This blog is a part of our Meet the Rustaceans series, where we invite the engineers and developers who use Rust on a regular basis to share their experiences and tell us about the amazing products that they are building using Rust here at Facebook. Look out for more interview blogs where we meet with many more engineers and hear their thoughts on this topic.

To learn more about Facebook Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Twitter and Facebook.

Interested in working in Production Engineering at Facebook? Check out our job postings on our Production Engineering career page here.