For today’s interview, we have Song Liu, a software engineer on the Kernel team at Meta. Song Liu recently presented at the Linux Plumbers Conference (LPC) on Kernel Live Patching at Scale.
This series highlights Meta Software Engineers who contribute to the Linux kernel. The Meta Linux Kernel team works with the broader Linux community to add new features to the kernel and makes sure that the kernel works well in Meta production data centers. Engineers on the team work with peers in the industry to make the kernel better for Meta's workloads and to make Linux better for everyone.
My name is Song Liu. I am a software engineer in the Kernel team, and my focus areas are BPF (Berkeley Packet Filter) development and kernel live patching.
I usually start the day replying to important emails in the mail list. Then, depending on the actual work I am taking on, I may write some code, debug some issues, write some slides and other tasks.
We have shipped kernel live patches to millions of servers in the fleet. Kernel live patching (KLP) makes it possible to apply quick fixes to a live Linux kernel, without having to shut down the workload to reboot a server. I am really proud of that work.
At Meta, we care about every fraction of performance. This makes us the first ones to find many smallish performance regressions in the upstream kernel. We have many interesting stories debugging and fixing these issues.
I presented our experience and work in using kernel live patching at Meta scale. Specifically, we described the new challenges on a big scale and discussed some ideas for future improvements.
We discussed how to build, deploy, and monitor KLP at Meta and some of the ways we’ve worked to improve KLP infrastructure. You can explore the presentation materials to learn more about the work we’re doing with KLP at scale.
I think being patient is very important, as it may take a long time for your work to land upstream.
A mentor with valuable experience is always helpful. Luckily, we have many experts on the team.
Image credit: Larry Ewing (email@example.com) and The GIMP for the original design of Tux the penguin.