Meet the Developers: Linux Kernel Team (Martin Lau)

For today’s interview, we have Martin Lau, a software engineer on the Kernel team at Meta. He works on the BPF (Berkeley Packet Filter) and Kernel Networking development.

This series highlights Meta Software Engineers who contribute to the Linux kernel. The Meta Linux Kernel team works with the broader Linux community to add new features to the kernel and makes sure that the kernel works well in Meta production data centers. Engineers on the team work with peers in the industry to make the kernel better for Meta's workloads and to make Linux better for everyone.

Tell us about yourself and what your typical day looks like as a kernel developer at Meta.

My name is Martin KaFai Lau. I have been with Meta for 9 and a half years. I joined the Meta Kernel team 6 years ago. I am focusing on BPF and networking development. Before that, I was on the Meta Traffic team doing HTTP, TLS and CDN work.

My typical day involves reviewing patches in the mailing list, supporting Meta use cases in production, exploring ideas with other networking teams and writing patches for upstream.

What have you been excited about or incredibly proud of lately?

Works that make kernel networking stack extensible by BPF. It also excites me to work on stackable workloads that need to impose bandwidth limitations per task, explore One-Way-Delay measurement with BPF and speed up a service start-up time from more than a minute down to 6 seconds.

Is there something especially exciting about being a Kernel developer at a company like Meta?

Scale. Being able to solve problems in Meta's production scale is very exciting. Kernel is the core piece used by all services, and it is the core interface between a computer’s hardware and its processes. Improvement in kernel will have a positive effect on all of them.

Scale is also about the number of internal users that have a lot of production experience, and each of them could have a diverge usage. It is so much faster to get feedback from different teams, and they can explain if something will work well in the real world or not. Their feedback also leads to new kernel development.

My coworkers are maintainers in different kernel subsystems that I am not familiar with. This is a great learning experience that is very valuable in Meta.

Tell us a bit about the topic you presented at the Linux Plumbers Conference (LPC) this year.

In the LPC presentation, I gave an overview of the BPF networking hooks in the kernel and how Meta uses them, the surprises I usually hear about from our internal users and what could be addressed in the future.

Over the years, BPF has grown considerably, so it can sometimes be difficult to navigate how BPF should be used in the networking stack. You can explore some of the resources from the talk on the event page “Overview of the BPF networking hooks and user experience in Meta.”

What are some of the misconceptions about kernel or OSS development that you have encountered in your career?

My previous life was user space only. I usually could quickly understand the big picture of the whole piece I was working on. I thought that would be hard in the kernel because the code base is large. More importantly, it is also tough to get help from upstream people.

I spent time reading and observing the mailing list and paid attention to how people work on the mailing list. It is impossible to get a grip on everything in the kernel. I shrink the scope, focus on one piece and fix something to gain credibility in the upstream. Then, I repeat the process to expand my knowledge space.

What resources are helpful in getting started in kernel development, and where can people follow your work?

The Linux kernel mailing list. Spend time reading the threads that you are interested in. Understand the concern and interests that the stakeholders usually have. Start with something small that solves a real production problem.

I am usually active in the BPF mailing list.

To learn more about Meta Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Twitter, Facebook and LinkedIn.

Image credit: Larry Ewing (lewing@isc.tamu.edu) and The GIMP for the original design of Tux the penguin.