On developing deep programming knowledge

Question

Occasionally I see questions about edge cases and other weirdness on Stack Overflow that are easily answered by the likes of Jon Skeet and Eric Lippert, demonstrating a deep knowledge of the language and its many intricacies, like this one:

You might think that in order to use a foreach loop, the collection you are iterating over must implement IEnumerable or IEnumerable<T>. But as it turns out, that is not actually a requirement. What is required is that the type of the collection must have a public method called GetEnumerator, and that must return some type that has a public property getter called Current and a public method MoveNext that returns a bool. If the compiler can determine that all of those requirements are met then the code is generated to use those methods. Only if those requirements are not met do we check to see if the object implements IEnumerable or IEnumerable<T>.

That's cool stuff to know. I can understand why Eric knows this; he's on the compiler team, so he has to know. But what about those who demonstrate such deep knowledge who are not insiders?

How do mere mortals (who are not on the C# compiler team) find out about stuff like this?

Specifically, are there methods these folks use to systematically root out such knowledge, explore it and internalize it (make it their own)?

Answer 1

First off, thanks for the kind words.

If you want to get a deep knowledge of C# it is undoubtedly an advantage to have the language specification, ten years of design notes, the source code, the bug database, and Anders, Mads, Scott and Peter just down the hall. I'm certainly fortunate, no question about it.

However, even without those advantages it is still possible to get a deep knowledge of the subject.

Back when I started at Microsoft I was working on the JScript interpreter that shipped with Internet Explorer 3. My manager at the time told me something that was some of the best advice I've ever gotten. He said that he wanted me to become the recognized expert at Microsoft on the syntax and semantics of the JScript language, and that I should go about this by seeking out questions on those aspects of JScript and answering them. Particularly answering the questions I didn't know the answers to, because those are the ones I would learn from.

Obviously StackOverflow and other public Q&A forums are like drinking from a firehose for that sort of thing. Back then, I read comp.lang.javascript and our internal Microsoft "JS User" forums religiously and followed my manager's advice: when I saw a question that was about the language semantics that I didn't know the answer to, I made it my business to find out.

If you want to do a "deep dive" like that, you've got to choose carefully. I to this day am remarkably ignorant of how the browser object model works. Since I have been concentrating on becoming the C# language expert these last years, I am remarkably ignorant of how the various classes in the base class libraries work. I am fortunate in that I have a job that prizes specific deep knowledge; if your job or your talents are more in line with being a generalist, going deep might not work for you.

Writing a blog is also tremendously helpful; by requiring me to explain complex topics to other people, I am forced to confront my own inadequate understanding of various topics all the time.

Answer 2

Having been on the "guru" side of the conversation once or twice, I can tell you that a lot of times what you perceive as "deep knowledge" of a programming language or system is often the result of the "guru" recently struggling for a month to solve the exact same problem. That's especially true on a forum where people can choose which questions they will answer. Even the likes of Jon Skeet and Eric Lippert had to learn hello world at one point. They pick up their knowledge one concept at a time, same as anyone else.

Answer 3

Paraphrasing Yogi Bhajan:

"If you want to learn something, read about it; if you want to understand something, write about it; if you want to master something, program it."

Programming is like the ultimate teaching challenge. Teaching computer do something requires, that you know your stuff really well -- or you will learn to master it.

For example, if you want to learn physics, write a physics engine. If you want to learn chess, program a chess game. If you want to learn deep C# knowledge, write a C# compiler (or some other tool).

Answer 4

As far as I know, the ways to learn this are:

• Read about it from someone like Eric Lippert
• Experience and then solve the problems first hand.

The second way may take much longer but will probably result in a deeper understanding (but not always).

Answer 5

I would say do the following:

After learning a relatively useful stack of languages (the ones you need for a real job) at the level where you can do most common tasks, stop learning more languages until you have studied at least one in depth. Part of the problem in our industry right now, in my opinion, is that people only learn the first 5-10% of the language before moving on to some other language. Once you have the capability to do most common tasks in a job, then start looking at one thing in depth. (You can go back to getting breadth after you get some depth, then go back and forth between the two.)

Volunteer for the more complex, harder tasks, the ones that make you have to go in depth to solve the problems. If there are none where you work, look for open source tasks to do or start working on a personal project that will make you have to go in depth. If your job has no interesting problems, consider looking for a more challenging job.

Read the advanced books on one language (for SQl Server for instance this would include reading about performance tuning and database internals) instead of the learn X in 30 days type of books.

Read the interesting questions here and other places where they are asked and try to solve some yourself. If you want to learn, try to solve some without reading the other answers first. Even if the question has already been answered, you will learn more if you find the answer yourself. You might even find a better answer than the one the question had.

Ask a few of the harder questions. Evaluate the answers you are given, don't just use them. Make sure to understand why the answer would or would not work. Use those answers as a starting place to research.

Find some good technical blogs from known experts in the field and read them.

Stop throwing away your knowledge after you are finished with it. Learn to retain. Most experts don't have to look up the common syntax. They don't have to reinvent the wheel every time they face a problem because they remember how they approached a simliar problem before. They can connect the dots and see how problem X that they did two years ago is similar to problem Y that they have right now (it amazes me how few people seem able to make connections like that). Consequently, they have more time available to spend researching more interesting subjects.

Answer 6

You can start by deeply studying the language specifications of the ones that you seek to be an expert off. For example:

• C#
• Java

Answer 7

Get Reflector or any other decompiler (since it's paying now), and start opening up some of the most used .NET libraries to learn how the internals work. Combined with a book like CLR via C# you'll get quite deep (deeper than most of us will go on their regular job).

Answer 8

I developed that kind of knowledge in C++ by hanging out in comp.lang.c++.moderated for a couple years, even though I wasn't really working that hard to code in it at that point. I'm not sure how guru I can say I am, though.

I think there's two kinds of knowledge that one can pick up about a programming language:

1. Knowing trivia about the language, and knowing how to avoid pitfalls.
2. Knowing how to solve problems effectively.

Number 2 can only be achieved by programming in the language and looking at other people's code, but number 1 can be achieved by taking lots of time to read about the language on its discussion forums, seeing what kinds of questions people ask, and what the answers are. StackOverflow is a good place for that too.

Answer 9

Deep knowledge and programming expertise means being comfortable at all abstraction levels. I.e.

• libraries and APIs
• language semantics
• compiler optimizations
• compiler internals and code generation
• runtime and garbage collector behavior
• architectural and instruction set issues

Everything I've seen in the last 15 years has shown that only if you really can get into the compiler and runtime do you have a chance of becoming deeply proficient. You may have to force yourself to take the step and start reasoning (and building) software at the next level of abstraction lower in the stack, but its the only way to expertise.

All we have is language for abstraction. You must understand how programming languages are designed and built to really know what the machine is doing.

Answer 10

Read The Fine Manual This is not particularly deep knowledge. It's published in the C# language specification section 8.6.4. You should get in the habit of at least skimming the specifications for the languages that you use, as well as skimming the documentation for all the builtin libraries.

Anyway, this not my idea of deep knowledge; it's just an uninteresting implementation detail. It could be more interesting if the designer explained why it was done in this more dynamic way, instead of just checking that the object implements Iterable.

Answer 11

ExScrutiny is important. 10000 hours spent doing something wrong is not helpful. You have to get your efforts judged, hopefully by the constructive leaders whose feedback you want, but be prepared to ignore many personality disorders who will try to make themselves feel good with personal attacks. (This happens least of all on Stack Exchange I find, it attracts downvotes.) Answering questions on Stack Exchange and working in Open Source gets you scrutiny. Writing your own blog and trying to find something non obvious and interesting to write about once a month, allowing comments to be posted, is also good.

(side benefit: if you actually are good, scrutiny will spread the word far better than claiming to be good. Which is why you've heard of some of the programmers mentioned in the article, and why you tend to ignore those who merely claim to be good in the absence of evidence)

Answer 12

How ? In most cases unintentionally, by accident. I think that depth of knowledge, when it is required in programming brings negative value.

The programming is already considered by some to be the most complex human activity ever. So if in addition to natural workload of domain knowledge, a thinker have to deal with unnecessary difficulty or pecularity, then this is a problem, not an advantage. The goal of programming (in 21st century, when nearly everything is done and overdone and several time forgotten) is to add code which adds more value than added complexity. It was always the goal. Well, but now the ratio of value to complexity should be even better than ever. What we really see in most languages and frameworks, is that this ratio is not improving, or is improving very slow.

If you find yourself wondering more about the tool, than the workpiece, time to look for better tool. The problem is so old, nearly as old as computer industry

Answer: mere mortals will be happy to never discover any deep secrets of their tools.

P.S: instead of being a complimentary knowledge, programming became too self serving occupation. I look up at electrical engineer with VB6 knowledge, or financial analyst with Excel, but rarely admire the pure software architect or developer like myself. Why, because we unfortunately never resolved the software crysis. May be future generations will.