For today’s interview, we have Martin Lau, a software engineer on the Kernel team at Meta. He works on the BPF (Berkeley Packet Filter) and Kernel Networking development.
This series highlights Meta Software Engineers who contribute to the Linux kernel. The Meta Linux Kernel team works with the broader Linux community to add new features to the kernel and makes sure that the kernel works well in Meta production data centers. Engineers on the team work with peers in the industry to make the kernel better for Meta's workloads and to make Linux better for everyone.
My name is Martin KaFai Lau. I have been with Meta for 9 and a half years. I joined the Meta Kernel team 6 years ago. I am focusing on BPF and networking development. Before that, I was on the Meta Traffic team doing HTTP, TLS and CDN work.
My typical day involves reviewing patches in the mailing list, supporting Meta use cases in production, exploring ideas with other networking teams and writing patches for upstream.
Works that make kernel networking stack extensible by BPF. It also excites me to work on stackable workloads that need to impose bandwidth limitations per task, explore One-Way-Delay measurement with BPF and speed up a service start-up time from more than a minute down to 6 seconds.
Scale. Being able to solve problems in Meta's production scale is very exciting. Kernel is the core piece used by all services, and it is the core interface between a computer’s hardware and its processes. Improvement in kernel will have a positive effect on all of them.
Scale is also about the number of internal users that have a lot of production experience, and each of them could have a diverge usage. It is so much faster to get feedback from different teams, and they can explain if something will work well in the real world or not. Their feedback also leads to new kernel development.
My coworkers are maintainers in different kernel subsystems that I am not familiar with. This is a great learning experience that is very valuable in Meta.
In the LPC presentation, I gave an overview of the BPF networking hooks in the kernel and how Meta uses them, the surprises I usually hear about from our internal users and what could be addressed in the future.
Over the years, BPF has grown considerably, so it can sometimes be difficult to navigate how BPF should be used in the networking stack. You can explore some of the resources from the talk on the event page “Overview of the BPF networking hooks and user experience in Meta.”
My previous life was user space only. I usually could quickly understand the big picture of the whole piece I was working on. I thought that would be hard in the kernel because the code base is large. More importantly, it is also tough to get help from upstream people.
I spent time reading and observing the mailing list and paid attention to how people work on the mailing list. It is impossible to get a grip on everything in the kernel. I shrink the scope, focus on one piece and fix something to gain credibility in the upstream. Then, I repeat the process to expand my knowledge space.
The Linux kernel mailing list. Spend time reading the threads that you are interested in. Understand the concern and interests that the stakeholders usually have. Start with something small that solves a real production problem.
I am usually active in the BPF mailing list.
Image credit: Larry Ewing (email@example.com) and The GIMP for the original design of Tux the penguin.