Production Engineering at Jane Street
Production engineers wear two hats: we divide our time between a support rotation in which we actively resolve live issues, and project work in which we build software to improve our support infrastructure.
About the role
Jane Street has an extensive tech landscape. By and large, our software engineers are responsible for both the development of their applications and the ongoing production support. But some applications, whether due to strict response requirements or some other characteristic of the production environment, require dedicated personnel to provide production support.
Enter production engineers.
Production engineers wear two hats: we divide our time between a support rotation in which we actively resolve live issues, and project work in which we build software to improve our support infrastructure. (You can read the job description to learn more.) Here we’ll describe both roles in more detail to help you decide whether it’s something you think you’d enjoy.
The support rotation
The primary service that production engineers provide the firm is a support rotation. We staff rotations for a given suite of applications in order to provide timely resolution to whatever operational issues come up throughout the trading day. We spend about a third of our time in these rotations, typically in weeklong chunks to remain uninterrupted during our project work.
At its most exciting, those issues are novel incidents, possibly arising as a result of a bad deployment, a new kind of trading, or something changing in the external world. Resolving these incidents requires a cool head, the ability to coordinate among multiple internal and external parties, and excellent debugging and problem-solving skills.
And at its most mundane, those issues are known bugs that we just haven’t fixed yet because we felt we had the capacity to take on some additional toil in order to get new features or new trading flows enabled.
Things we enjoy about support
• Quick wins: Writing software can take weeks of work to produce an unobservable gain. On the other hand, resolving production issues produces lots of observable gain really quickly—getting these quick wins can feel great.
• Debugging can be fun: Debugging hairy problems, digging into unfamiliar systems, combining multiple incomplete data sources to build a full story… if that all sounds like fun to you, then you might enjoy being a production engineer!
• Great learning: You’ll get exposure to a wide variety of Jane Street systems and see how different teams do things, which can inform how you build your own software.
• It’s lively: It’s definitely not sitting down with your headphones on, hammering out code. While on the rotation, you’re more likely to be up on your feet, coordinating with multiple teams (maybe including external parties) to resolve an issue. Again, a lot of people find this fun!
Things to keep in mind
We enjoy support, but it’s not for every engineer. If you want to spend as much time as possible with your headphones on and crushing out code, this probably isn’t the role for you. Specifically, you should know that your time on support can be:
• Interrupt driven: We don’t expect people to get any project work done while they participate in the rotation, because the frequent interruptions make that impossible.
• Repetitive: We mostly try to be constantly dealing with novel issues—if we’re dealing with the same issues that we dealt with a year ago, then we aren’t doing a good enough job of automating things. But inevitably, issues will recur. When that happens too often, it can be frustrating to deal with an issue that you know how to resolve and just haven’t had the time to fix yet.
• Sometimes stressful: Dealing with incidents can be fun, but there’s no denying that it’s stressful to have production trading potentially depend on your ability to resolve a live issue.
These aren’t negatives to everyone. If the idea of writing code 100% of the time sounds monotonous to you, and you like solving live problems, you’ll probably enjoy the variety and excitement of support (we all do!).
Project Work
When we’re not participating in the rotation, we write software1 to make the support rotation better. This generally takes one of two forms: working directly with application developers on the applications that we support or working on generic support tooling.
Application work
Many production engineers work closely with application developers on the specific applications that we help support, but with a focus on improving the reliability and supportability of those applications.
The work itself, whether it’s improving noisy alerts or making recovery from incidents easier or something else, brings obvious immediate benefits.
But we think it brings longer-term benefits as well. Production engineers, by supporting a wide variety of applications, often have a broader perspective than individual application developers on what makes applications easy to support in production. Conversely, application developers typically understand their own applications in a deeper way than production engineers otherwise would. We believe working together is the best way to build systems that share the best practices developed across the firm while still handling the unique challenges posed by the specific problem space.
Infrastructure Tools
Production engineers are also responsible for building a lot of the tooling that’s required to run an effective support rotation. We work on alerting libraries, toil tracking and visualization, logging libraries, and more, much of which is used broadly across the firm.
What it’s not
In the wider world, the term Production Engineering is sometimes used to refer to a broader concept, including systems work like performance tuning of Linux platforms, datacenter management, and so on. If you’re interested in that, you may be interested in applying as a Linux Engineer.
At some firms, it also means deploying other people’s code. We don’t have a dedicated role or team for that at Jane Street—teams are responsible for deploying their own code.
1. Just like everyone else at Jane Street, we write almost all of our software using OCaml. And just like pretty much everywhere else at Jane Street, we don’t expect you to know any OCaml coming in—we’ll teach you what you need to know.
