Sr. Software Engineer, Linkerd at Buoyant

Posted on: 06/09/2022

Location: (REMOTE)

Original Source

Tags: grpc linkerd github rust prometheus kubernetes containers bash

Hi, we're Buoyant. ------------------ We build [Linkerd](—the lightest, fastest, simplest service mesh for Kubernetes.Linkerd is an open source project. It has a thriving community of adopters and it powers the production systems of companies around the world. Companies like Microsoft, Nordstrom, Timescale, Expedia, and many, many more use Linkerd to add critical security, reliability, and observability features to their Kubernetes applications. We’re a small company with an incredibly outsized impact on the world. We’re remote-first and fully distributed with team members all over the world. Our competition is fierce: every day, we take on some of the largest companies in the world with essentially infinite dollars and infinite engineers to devote to their competing projects. Our secret weapon? Focus, and a deep sense of empathy for our users. Working here is not for the faint of heart. ### What's Linkerd? Imagine taking ultralight, ultrafast L7 "micro-proxies" written in [Rust](, sticking them next to other people's applications running on other people's clusters, and having them mediate all TCP (and in the future, UDP) communication to and from these apps—no matter what it is. Imagine these proxies upgrading connections from HTTP/1 to HTTP/2, initiating and terminating mutual TLS, retrying requests, emitting fine-grained metrics, issuing CSRs to the local CA every n hours, and so on. And imagine the end user blissfully operating 10,000 of these proxies, as a whole, with an easy-to-use CLI and API. That's a service mesh. (The best way to learn more is to [try it yourself]( If you can run Kubernetes, you can get Linkerd up and running in 5 minutes.) Under the hood Linkerd is extremely sophisticated and deeply technical, but the key to the entire project is simplicity. It's extremely difficult to make a simple service mesh, and very easy to make a complex one. (Heck, look at pretty much any of the other options.) And simplicity in this case means operational simplicity: it doesn't mean that you get a one-click install wizard, it means that whenever Linkerd is running, you can understand what it's doing and predict its behavior. It means no leaky abstractions, no complex tuning, no hidden gotchas. We spend a lot of time and energy in ensuring Linkerd is simple—it's our central promise to our users. Linkerd is a graduated-tier project of the Cloud Native Computing Foundation, just like Kubernetes, Prometheus, and other defining projects of the cloud native space. It's written in Rust and Go; Go powers (most of) Linkerd's control plane. Rust powers Linkerd2-proxy, the source of Linkerd's true power. We're the only service mesh that uses something like Linkerd2-proxy, and it's a tremendous differentiator. On the commercial side, we make Buoyant Cloud, a cloud-based management system for Linkerd. Buoyant Cloud makes Linkerd stress-free to operate. You connect your existing deployments, and Buoyant Cloud monitors their health and alerts you if anything looks weird. It takes on the toil of upgrades, data plane management, and more. In short: Buoyant Cloud turns Linkerd into a managed service, even though it's running on your own clusters. ### What are you looking for? In this role, we're looking for software engineers who want to work on Linkerd as a whole. Linkerd is built in a combination of Go (in the control plane) and Rust (in the proxy, and recently in the control plane as well). It's not trivial code: Linkerd sits at the intersection of Kubernetes, the network, and the user's application, and any of them can do very weird things and Linkerd needs to keep working. Not only does it need to keep working, it needs to have all the features that people need AND it needs to improve their security while not introducing new security issues AND it needs to be operationally simple AND performant AND low resource. I has to handle all of that gracefully, without requiring complex tuning, without having unpredictable resource usage, and while running in a container on unpredictable hardware with an unpredictable network. We're a small company and everyone wears a lot of hats. You will need to write some Go. You will need to write some Rust. You will need to write some bash scripts and fight with CI. You will need to interact with actual users and treat them nicely even if they ask silly questions. Linkerd is an open source project, and it feels like one: we have a lot of users, a continuous stream of GitHub issues and Slack messages asking for help or describing problems or asking when something is going to be ready. The cloud native space is rife with buzz, FOMO, and distraction. You need to be able to be productive in that environment. We interact with our Linkerd users in the [community Slack]( and we try to engage with them authentically while not being distracted. You don't have to deliver keynotes at Kubecon (unless you want to!), but participating in the community is part of your job. We’re a team, not a family, but we have families and are the kind of place where work doesn’t get in the way of that. There will be crunch times, but work life balance is important and our p95 work times are pretty good for a startup. This is an extremely technical project that pushes the boundaries of modern network software. If that sounds fun, this just might be the job for you. ### You have: + Eight or more years of professional software engineering experience developing high-performance, low latency networked services + Expert-level understanding of Linux operating system fundamentals and network programming + The ability to navigate complex software engineering tradeoffs between performance, maintainability, future-proofing, end-user experience, and feature completeness + Excellent written communication skills + The ability to work with in a distributed team with members across with different timezones + Familiarity with Linkerd, Kubernetes, gRPC, and containers + A weird but adorable love of software infrastructure ### In this role, you will: + Write mission-critical, high-performance open source code in Rust and Go + Design and implement complex features at the intersection of network programming, operating systems, Kubernetes, and cloud native deployment models, in collaboration with other team members + Give and receive critical feedback on design and code while maintaining a supportive and friendly environment. + Ensure performance and correctness of Linkerd across multiple dimensions, including traffic load, protocol choice, resource constraints, and product requirements + Develop high-scale reliable test tooling and environments, drawing from experience working on production systems + Solve problems aka fight with CI/CD tools, Linux systems, networking, and the rest of the complex ecology in which Linkerd lives ### You are: + Low-ego, collaborative, and results-oriented + Able to give and receive constructive feedback + Willing to learn from and mentor teammates + Passionate, empathetic, and kind Buoyant is a fully distributed, remote-first company. Applicants from anywhere in the world can apply. However, working hours must overlap at least partially with US or EU timezones.