Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am enamored by reliable distributed systems and want to spend my career improving them rather than merely using them to write the same CRUD applications over and over.

What experience will convince an employer to let me work on their systems project? (Pick any example, say, CouchDB.)

  • Presumably writing my own system is the best example, but the scope of that is relatively large and pragmatically speaking it doesn't make a ton of sense to re-invent the wheel.

  • Learning the codebase of an existing system and working up to submitting patches seems like a long road to only marginal impact on a project, with little assurance of job payoff.

  • Most employers (as far as I can tell) want you busy working on their high-level app and not spending time improving the infrastructure.

Are these impressions accurate? How do other people get started working on these systems?

EDIT: I am indeed willing to start at the bottom and labor for a long time. Are there specific projects/tasks in the distributed systems space that are suitable for a beginner? E.g. it seems like distributed message queues are a good starter project and are not 100% saturated across all platforms yet.

Any thoughts greatly appreciated. Thank you!

share|improve this question

closed as off-topic by MichaelT, Kilian Foth, GlenH7, Dan Pichelman, durron597 Jun 5 '15 at 4:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking career or education advice are off topic on Programmers. They are only meaningful to the asker and do not generate lasting value for the broader programming community. Furthermore, in most cases, any answer is going to be a subjective opinion that may not take into account all the nuances of a (your) particular circumstance." – Community, Kilian Foth, GlenH7, Dan Pichelman, durron597
If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 2 down vote accepted

Generally speaking, if it ain't broke, it ain't gonna be fixed.

If your company already has systems in place that solve this and they don't see a problem with it, chances are they're not going to want to pay you to fix it, which doesn't make them any money (relative to those applications that people use, with those boring CRUD ops ;)).

If your company has a department that does this, then have a chat with your manager. Your manager should be well aware of your career aspirations and help you move in that direction (if he/she doesn't, then you might want to consider looking elsewhere). If there is such a department at your current employer, find someone to chat with who's currently there (preferably in a senior role). Take him/her out for coffee.. Show your interest.. Find out what it would take to get you from where you are to there.

If your company doesn't have such a department, then do some studying. Read up on design patterns and algorithms used in that domain of development. Find some OS stuff that you can help contribute to. Even if you don't make a major overhaul to the system, it still shows your passion for the subject and will give you some real life visibility into development practices.

share|improve this answer

You have two distinct but related challenges: learning about "reliable distributed systems" (eventually you will discover that is an incomplete specification within a large and complex field); and getting a job in that field at some point in your education.

Two activities are crucial to the education challenge: becoming familiar with the pertinent literature; and experimental implementations. I infer that your interest is at the infrastructure (e.g., middleware) level rather than at the application level. There are fewer infrastructure experts than application experts, just as for operating systems. OS and middleware experts need the expertise to make programming easier for their users.

I am going to assume that you want (as you should) to begin to understand the principles of distributed systems, not just programming techniques. Most of the foundational literature is in a relatively small number of research journals and conference proceedings, and in a relatively small number of books (textbooks and research monographs).

The research literature is essential but oriented toward subject matter experts, so you should start with some textbooks. Unfortunately, IMHO currently there are no really good distributed (especially reliable distributed) textbooks. Some are too narrow and deep, some are too out of date, some are too shallow. When I taught a distributed systems course at CMU, I created balanced contemporary lectures augmented by synthesizing a "book" from a number of journal and conference papers, and I would do so again today. But given the need to purchase a book (or two), with considerable reluctance I recommend Ghosh "Distributed Systems: An Algorithmic Approach," plus Coulouris et al. "Distributed Systems: Concepts and Design (5th Edition)."

For the next level of detail, I suggest focusing on publications in the ACM and IEEE Computer Society Digital Libraries. You can search them using Google Scholar or at their web sites. Depending on your motivation and finances, you might consider paying for PDF downloads from these libraries. I particularly recommend that you explore the Proceedings of the ACM Symposium on the Principles of Distributed Computing, and the Proceedings of the IEEE International Conference in Distributed Computing Systems. Amazon has a vast collection of in-depth books on almost every aspect of reliable distributed systems.

Soon after you begin your study of the principles in the literature, you should do some experimental design and implementation.

The most accessible way is to use an open source version of Linux running on at least three PC's connected by an Ethernet. For your purpose, the PC's can be very inexpensive out of date units. It is important that you find a community of interest around the Linux version so you are not on your own. That community should be academically based, not associated with some Linux product, to give you freedom and support to experiment with your own version.

I have not looked around recently for candidates, as you should. But I know one very well: Virginia Tech's ChronOS, ChronOS has a feature important to your interest in reliable distributed systems: it includes an unusual emphasis on some state of the art distributed fault management concepts and techniques. It also emphasizes real-time, which may be peripheral to your interests, except that performing distributed fault management in real-time is exceptionally challenging, and would provide you some rare subject matter expertise that conceivably could help in your search for employment. ChronOS is led by Prof. Binoy Ravindran,, Although the ChronOS community of interest is not as large as that of some other Linux versions, it is likely to be flexible and supportive of your desire to study it and make changes to your own version. It may be possible that you create features that Binoy wants to include in the baseline ChronOS.

As for your second major challenge, to get a job in the field of reliable distributed systems, there are the obvious generic ways to search for opportunities. But if you are a budding subject matter expert on reliable distributed system infrastructures, though that is a niche, it is a very important one (e.g., more people program microprocessors than design them, but microprocessor designers are obviously essential). There are vendors of middleware and so-called enterprise system buses, such as Oracle, TIBCO, etc. (you can locate them easily). Note that there are application domains that at least sometimes if not frequently create their own application-specific middleware or modify commercial products: defense (extensively), telecommunications, and industrial automation.

share|improve this answer
Thank you, I appreciate the thoughtful answer! Will definitely check out the publications and projects you mentioned. – elliot42 Sep 26 '11 at 20:00
Thanks your advice... – Sayakiss Mar 24 at 9:42

What you are looking for is a niche job.

The only way to get one of those is like any other really: apply for a job that fits the description. Of course since you have little experience in the design and building of such systems you'd have to either run for an entry level job, or otherwise leverage your experience if any of it could be of interest to that particular employer.

The bottom line is that while it can be difficult to change directions in your career, particularly for some highly specialised area, it is possible. Nobody starts out a subject matter expert in anything, you gain that by experience. But when you switch from one to the other you generally, though not necessarilly, must accept to start down a rung or two before you regain your previous level or status.

EDIT: some more thoughts...

@Elliot42: Read up on capacity, performance, scalability, resilience, robustness. Know about deadlock, livelock, race conditions, synchronous processes, asynchronous processes, messaging systems, sockets, mutexes... Best way to get started on all of those is to practice by designing and building multi-tasking applications or systems. Look for known problems to implement and check up on known solutions (such as a multi-user game or a message board for instance for which you implement low level communications). Once you have built a single, self contained multi tasked application (however you implement that, whether with threads, a multi-tasking engine, a real-time engine, separate processes - it does not matter), build a second one and let them communicate with one another. Try a client-server with multiple clients, try a peer to peer, try a master-slave redundant application. You don't have to do it all, but sample real problems, read up on the others, then apply for a job.

share|improve this answer
Good point, thanks. I guess more specifically I am wondering if anyone know what the entry-level tasks in the distributed systems world are, so I can go get some experience in those. – elliot42 May 3 '11 at 22:30
@elliot42, see update answer. – asoundmove May 4 '11 at 3:42

My pick would be to major in distributed systems at university, get a good hang of the ideas in the subject, find an employer suitable for you and work with them to your heart's content.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.