A coworker asked me: "I really enjoyed CS systems and theory classes in school, but my industry experiences have always been very run-of-the-mill web dev-y / not related to the stuff I studied — how can I combine them / keep aspects of both in my intellectual and professional life?"
I rambled for a bit and they said the answer was helpful, so reposting here:
Unless you get very lucky in finding a project with an existing research-focused champion (a bit more common at less-glamorous companies that do not have Bevi water machines), you will almost never be told, “here’s our project X, we think theories Y/Z might be applicable, let’s read up and figure out whether/how they apply.” Most people have no idea that Y/Z exist, or that X is even an instance of a broader class of problems with known solutions/approaches. Everyone thinks their problem is unique. Even tons of smart, intellectually curious people will just dive in and try to come up with their own “good-enough” ad-hoc solution or overly-clever implementation, rather than looking for related work, because we do not teach engineers how to find related work.
This means that in practice, 99% of opportunities to apply theory to engineering requires *you* to recognize cases where theory could provide a significant improvement over the current direction. Then you need to successfully intervene to communicate this/push for a more principled approach. Plenty of people are receptive to doing things better, but literally don’t realize things *could* be better.
To be good at recognizing those opportunities, you need to *already know what is out there* (what the solution landscape looks like, which problems have already been solved decades ago, who’s an expert on what today), and you need to be able to *identify parallels between superficially disparate problems* (e.g. reductions like network flow, recognizing that elevators and serial disk scanning use the same algorithm, figuring out that a particular build process sort of requires implementing a quine, or that the Confluence importer can be implemented as a compiler). While the dream is obviously for your job to expose you to cool ideas in computer science, in practice *you* are exposing your job to these ideas.