Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

As a "new" programmer (I first wrote a line of code in 2009), I've noticed it's relatively easy to create a program that exhibits quite complex elements today with things like .NET framework for example. Creating a visual interface, or sorting a list can be done with very few commands now.

When I was learning to program, I was also learning computing theory in parallel. Things like sorting algorithms, principles of how hardware operates together, boolean algebra, and finite-state machines. But I noticed if I ever wanted to test out some very basic principle I'd learned in theory, it was always a lot more difficult to get started because so much technology is obscured by things like libraries, frameworks, and the OS.

Making a memory-efficient program was required 40/50 years ago because there wasn't enough memory and it was expensive, so most programmers paid close attention to data types and how the instructions would be handled by the processor. Nowadays, some might argue that due to increased processing power and available memory, those concerns aren't a priority.

My question is if older programmers see innovations like these as a godsend or an additional layer to abstract through, and why might they think so? And do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

23  
Joel Spolsky's article from 2002 is relevant: joelonsoftware.com/articles/LeakyAbstractions.html As phrase/formulated, however, this question might end up being considered primarily opinion-based. –  BrianDHall Jan 14 at 16:58
    
I do lament the lack of more simple options for exploring very basic programming techniques. –  GrandmasterB Jan 14 at 17:20
1  
This is relevant. Sort of. (I mean, I hope that image is a joke, but with some of what gets on to StackOverflow...) –  Izkata Jan 14 at 21:59
1  
It has its pros and cons. It opens the programming world to a lot of new programmers, who don't need as high skill to succeed - contrary to what some people might say, that's a good thing. And we're still writing efficient programs, that never changed - it's just that the goals have changed. Saving a byte on a year in date is no longer a good thing - the memory difference is usually unlikely to make a difference, and using eg. two bytes is more flexible and future-proof. Cost of programmers vs. cost of SW and HW is also a significant factor. The demand for new software is huge. Coders are few. –  Luaan Jan 16 at 8:22
1  
40/50 yr timescale is wrong. Memory efficient programming was still critically important in 16 bit Windows (less than 20 years ago) and (unfortunately) in most embedded/robotics today. –  david.pfx Jan 21 at 2:09

10 Answers 10

it just isn't necessary because of the increased amount of processing power and memory available.

Having cheap memory, enormous disks and fast processors isn't the only thing that has freed people from the need to obsess over every byte and cycle. Compilers are now far, far better than humans at producing highly optimized code when it matters.

Moreover, let's not forget what we're actually trying to optimize for, which is value produced for a given cost. Programmers are way more expensive than machines. Anything we do that makes programmers produce working, correct, robust, fully-featured programs faster and cheaper leads to the creation of more value in the world.

My question though is how do people feel about this "hiding" of lower-level elements. Do you older programmers see it as a godsend or an unnecessary layer to get through?

It is absolutely necessary to get any work done. I write code analyzers for a living; if I had to worry about register allocation or processor scheduling or any of those millions of other details then I would not be spending my time fixing bugs, reviewing performance reports, adding features, and so on.

All of programming is about abstracting away the layer below you in order to make a more valuable layer on top of it. If you do a "layer cake diagram" showing all the subsystems and how they are built on each other you'll find that there are literally dozens of layers between the hardware and the user experience. I think in the Windows layer cake diagram there's something like 60 levels of necessary subsystems between the raw hardware and the ability to execute "hello world" in C#.

Do you think younger programmers would benefit more learning low-level programming BEFORE exploring the realms of expansive libraries?

You put emphasis on BEFORE, so I must answer your question in the negative. I'm helping a 12 year old friend learn to program right now and you'd better believe I'm starting them in Processing.js and not x86 assembler. If you start a young programmer in something like Processing.js they'll be writing their own shoot-em-up games in about eight hours. If you start them in assembler they'll be multiplying three numbers together in about eight hours. Which do you think is more likely to engage the interest of a younger programmer?

Now if the question is "do programmers who understand layer n of the cake benefit from understanding layer n - 1?" the answer is yes, but that's independent of age or experience; it's always the case that you can improve your higher level programming by understanding better the underlying abstractions.

share|improve this answer
12  
+1 fun citation: Dunbars Number is a good example (there are others) of studied cognitive capacity quotients that can be seen across many people showing that we do have a fixed mental space. Abstracting multiple things into singular generalizations is the only way we can cohesively increase the number of things in our mental space, and that's what higher level programming seeks to take advantage of. –  Jimmy Hoffa Jan 14 at 17:25
16  
@Euphoric: Your question makes sense but where do you stop? Suppose I say "well, instead of learning Processing.js let's learn how to write video games in C++ using DirectX". OK, fine. Now what stops you from saying "isn't it going to create problems if they want to do something less abstract?" and so maybe we want to start with how to write a graphics card driver. But then you ask the question again, and before you know it, we're entering machine code into an Altair with toggle switches. You've got to pick a level of abstraction somewhere! –  Eric Lippert Jan 14 at 17:45
28  
@Euphoric: Would you make the same argument to, say, accounting? We don't need more people who can keep the simple books for a new small business; we need GREAT, world-class accountants. If introductory accounting courses are so difficult that they scare away people who merely aspire to be productive, workmanlike accountants, GOOD. We don't need those people in the accounting industry! How about pianists? If introductory piano lessons scare away people who are not going to be GREAT pianists, that's good; we only want great pianists in the world. Does something seem wrong with this argument? –  Eric Lippert Jan 14 at 18:17
22  
@Euphoric: The short answer is GOOD HEAVENS YES we need more decent programmers! We need more programmers at every level of ability and experience. The world runs on software. The more people who have any understanding of how to make their world work, the better. –  Eric Lippert Jan 14 at 18:19
6  
@Euphoric (and others) - we're kind of hitting the limit regarding constructiveness of comments. Please join us in Programmers Chat if you would like to continue this conversation. –  GlenH7 Jan 14 at 18:36

I had ideas on this subject, and I put them into a book 20 years ago. It's long out of print, but you can still get used copies on Amazon.

One simple answer to your question is as old as Aristotle: Nature abhors a vacuum. As much as machines have gotten faster and bigger, software has gotten slower and bigger.

To be more constructive, what I proposed was that information theory, and its direct relevance to software, be part of computer science education. It is only taught now, if at all, in a very tangential way.

For example, the big-O behavior of algorithms can be very neatly and intuitively understood if you think of a program as a Shannon-type information channel, with input symbols, output symbols, noise, redundancy, and bandwidth.

On the other hand, the productivity of a programmer can be understood in similar terms using Kolmogorov information theory. The input is a symbolic conceptual structure in your head, and the output is the program text that comes out through your fingertips. The programming process is the channel between the two. When noise enters the process, it creates inconsistent programs (bugs). If the output program text has sufficient redundancy, it can permit the bugs to be caught and corrected (error detection and correction). However, if it is too redundant it is too large, and its very size, combined with the error rate, causes the introduction of bugs. As a result of this reasoning, I spent a good part of the book showing how to treat programming as a process of language design, with the goal of being able to define the domain-specific-languages appropriate for a need. We do pay lip service to domain-specific-languages in CS education but, again, it is tangential.

Building languages is easy. Every time you define a function, class, or variable, you are adding vocabulary to the language you started with, creating a new language with which to work. What is not generally appreciated is that the goal should be to make the new language a closer match to the conceptual structure of the problem. If this is done, then it has the effect of shortening the code and making it less buggy simply because, ideally, there is a 1-1 mapping between concepts and code. If the mapping is 1-1, you might make a mistake and code a concept incorrectly as a different concept, but the program will never crash, which is what happens when it encodes no consistent requirement.

We are not getting this. For all our brave talk about software system design, the ratio of code to requirements is getting bigger, much bigger.

It's true, we have very useful libraries. However, I think we should be very circumspect about abstraction. We should not assume if B builds on A and that is good, that if C builds on B it is even better. I call it the "princess and the pea" phenomenon. Piling layers on top of something troublesome does not necessarily fix it.

To terminate a long post, I've developed a style of programming (which sometimes gets me in trouble) where

  • Invention is not a bad thing. It is a good thing, as it is in other branches of engineering. Sure it may be creating a learning curve for others, but if the overall result is better productivity, it is worthwhile.
  • Haiku-style minimalist code. That especially goes for data structure design. In my experience, the biggest problem in software these days is bloated data structure.
share|improve this answer
9  
This sounds very much what Chuck Moore (inventor of Forth) has always advocated. For instance, from Chuck Moore's Comments on Forth, "I do not think it is intrinsic in the nature of software that it has to be large, bulky, buggy. It is in the nature of our society. ... There is so much economic motivation for producing this ... bloatware, that I kind of feel irresponsible in standing up and saying the emperor has no clothes.". –  Peter Mortensen Jan 14 at 19:32
3  
@PeterMortensen: Agreed. It's lonely. There's a word for that - Cassandra complex. –  Mike Dunlavey Jan 14 at 19:45
2  
While writing code artifacts to "extend" languages is not a difficult undertaking, making a good API that accurately and intuitively mirrors the problem domain is. –  Robert Harvey Jan 14 at 20:43
3  
@MikeDunlavey: BTW: You are also the "no-profiler" guy (this is meant in a positive way). A few months ago, I again used your technique in the real world to swiftly reduce the load time of a document file from typically 25 seconds to 1 second (a load time the user directly experiences). It took a few iterations, and 10-20 samples in all iterations were more than sufficient. The performance problems were of course in unexpected places. –  Peter Mortensen Jan 14 at 22:51
2  
@Izkata and Peter: Yeah, I'm that oddball. FWIW, I put up a couple (extremely amateur) videos, in hopes of making it easier to understand. Random Pausing. Differential Execution. Cheers. –  Mike Dunlavey Jan 14 at 22:56

High-level abstraction is essential to achieving ongoing progress in computing.

Why? Because humans can only hold so much knowledge in their heads at any given moment. Modern, large scale systems are only possible today because you can leverage such abstractions. Without those abstractions, software systems would simply collapse under their own weight.

Every time you write a method, you're creating an abstraction. You're creating a bit of functionality that's hidden behind a method call. Why do you write them? Because you can test the method, prove it works, and then invoke that functionality any time you want just by making the method call, and you don't have to think anymore about the code that's inside that method.

In the early days of computing, we used machine language. We wrote very small, bare metal programs with intimate knowledge of the hardware we were writing them for. It was a painstaking process. There were no debuggers; your program usually either worked, or it crashed. There was no GUI; everything was either command-line or batch process. The code you wrote would only work on that particular machine; it would not work on a machine with a different processor or operating system.

So we wrote high-level languages to abstract all of that detail away. We created virtual machines so that our programs could be portable to other machines. We created garbage collection so that programmers wouldn't have to be so diligent about managing memory, which eliminated a whole class of difficult bugs. We added bounds checking to our languages so that hackers couldn't exploit them with buffer overruns. We invented Functional Programming so that we could reason about our programs in a different way, and rediscovered it recently to take better advantage of concurrency.

Does all this abstraction insulate you from the hardware? Sure it does. Does living in a house instead of pitching a tent insulate you from nature? Absolutely. But everyone knows why they live in a house instead of a tent, and building a house is a completely different ball game than pitching a tent.

Yet, you can still pitch a tent when it is necessary to do that, and in programming, you can (if you're so inclined) still drop down to a level closer to the hardware to get performance or memory benefits that you might not otherwise achieve in your high-level language.


Can you abstract too much? "Overtake the plumbing," as Scotty would say? Of course you can. Writing good API's is hard. Writing good API's that correctly and comprehensively embody the problem domain, in a way that is intuitive and discoverable, is even harder. Piling on new layers of software isn't always the best solution. Software Design Patterns have, to some degree, made this situation worse, because inexperienced developers sometimes reach for them when a sharper, leaner tool is more appropriate.

share|improve this answer
1  
+1, although I think you got the history of functional programming backwards (the lambda calculus predates electronic computers, and many functional languages predate concurrent programming). –  amon Jan 14 at 17:35
1  
I added a small clarification. –  Robert Harvey Jan 14 at 17:36
2  
"In the early days of computing, we used machine language. We wrote very small, bare metal programs with intimate knowledge of the hardware we were writing them for." Some of us in embedded programming are occasionally still doing that, on 8-but microcontrollers that have less than 1K of program memory, 64 bytes of RAM, and cost around a quarter. No C compiler there. (But I usually work with 32-bit microcontrollers with typically 1/2 MB of program space.) –  tcrosley Jan 15 at 0:17
1  
@gnat: Thanks. I guess you got the reference. –  Robert Harvey Jan 15 at 23:50

I feel that high level programming has many advantages and is an essential part of a programming language. One of the reasons why Java become successful was that it has a comprehensive library. You achieve more with less code - just call a predefined function.

We can now distinguish programming language users from programming language writers (compiler writers). We leave the optimisations to compiler writers. We focus more on maintainability, reuse, etc

share|improve this answer

Really good training involves both extremes, as well as a bridge between.

On the low-level side: how a computer executes code from the ground up*, including knowledge of assembly language and what a compiler is doing.

On the high-level side: general concepts, e.g. using associative arrays, closures, etc. without having to waste time worrying about how it works under the hood.

IMHO everyone should have experience with both, including their drawbacks, and a taste of how to get from low-level concepts to high-level concepts. Love associative arrays? Great, now try using them on an 50-cent embedded processor with 1kB of RAM. Like writing fast code using C? Great, now you have three weeks to write a web app; you can spend your time dealing with data structures and memory management using C, or you can spend your time learning a new web framework and then implement the web app in a few days.

As far as the complexity aspect of it goes: I do think it is too easy these days to make complex systems without a clear understanding of the cost of doing so. As a result, we have, as a society, built up a huge amount of technical debt that bites us from time to time. It's like earthquakes (just the cost of living near a geological fault, right?), only it's gradually getting worse. And I don't know what to do about it. Ideally we'd learn and get better at managing complexity, but I don't think that's going to happen. A responsible engineering education needs to include a lot more discussion of the consequences of complexity than most of our universities are currently providing.


* and, anyway, where is the "ground" in how a computer executes code? Is it assembly language? Or computer architecture? Or digital logic? Or transistors? Or device physics?

share|improve this answer

The increase in the complexity of systems is relentless, oppressive and ultimately crippling. For me as an older generation programmer, it is also bitterly disappointing.

I've been programming for well over 40 years, having written code in 50-100 different languages or dialects, and become expert in 5-10. The reason I can claim so many is that mostly they're just the same language, with tweaks. The tweaks add complexity, making every language just a little different.

I have implemented the same algorithms innumerable times: collections, conversions, sort and search, encode/decode, format/parse, buffers and strings, arithmetic, memory, I/O. Every new implementation adds complexity, because every one is just a little different.

I wonder at the magic wrought by the high flying trapeze artists of the web frameworks and mobile apps, at how they can produce something so beautiful in such a short time. Then I realise how much they don't know, how much they will need to learn about data or communications or testing or threads or whatever before what they do becomes useful.

I learnt my craft in the era of fourth generation languages, where we genuinely believed that we would produce a succession of higher and higher level languages to progressively capture more and more of the repetitive parts of writing software. So how did that turn out, exactly?

Microsoft and IBM killed that idea by returning to C for writing apps for Windows and OS/2, while dBase/Foxpro and even Delphi languished. Then the web did it again with its ultimate trio of assembly languages: HTML, CSS and JavaScript/DOM. It's been all downhill from there. Always more languages and more libraries and more frameworks and more complexity.

We know we should be doing it differently. We know about CoffeeScript and Dart, about Less and Sass, about template to avoid having to write HTML. We know and we do it anyway. We have our frameworks, full of leaky abstractions, and we see what wonders can be done by those chosen few who learn the arcane incantations, but we and our programs are trapped by the decisions made in the past. It's too complicated to change or start over.

The result is that things that ought to be easy are not easy, and things that ought to be possible are nearly impossible, because of complexity. I can estimate the cost of making changes to implement a new feature in an established code base and be confident I'll be about right. I can estimate, but I can't justify it or explain it. It's too complicated.

In answer to your final question, I would strongly advise younger programmers to start as high on the layer cake as they possibly can, and only dive down to the lower layers as the need and desire provide the impetus. My preference is for languages with no loops, little or no branching and explicit state. Lisp and Haskell come to mind. In practice I always finish up with C#/Java, Ruby, Javascript, Python and SQL because that's where the communities are.

Final words: complexity is the ultimate enemy! Beat that and life becomes simple.

share|improve this answer
    
My 30+yrs of programming have taught me to use the highest level language available that will do what needs to be done and still allow the use of lower level languages when required. Easiest environment for that is shell scripting which can invoke modules written in any language. The hard part is breaking the dominant mindset that all functionalities of a project have to be implemented in the same language. –  DocSalvage May 23 at 4:43
    
@dicsalvage: I agree, and my big disappointment is the lack of ever higher levels. What awk could do in 10 lines in the 1980s now take 15 lines in Ruby or Python, but I look for something to do it in 3. And the locked down environments on phones mean the same thing takes 50 in Java or Objective C. No shell scripts there! –  david.pfx May 23 at 5:20
    
Google "bash for android" shows a lot of folks working on ports. There's also versions of Python like Kivy –  DocSalvage Jun 13 at 12:48
    
@DocSalvage: Sooner or later the phone environment will join civilisation (as we know it). My complaint is the time taken to do over and over again the things that seem to have been finished. We keep having to go back to laying foundations and brickwork and drainage and shacks when I want to be building skyscrapers. –  david.pfx Jun 13 at 14:12

My question though is how do people feel about this "hiding" of lower-level elements. Do you older programmers see it as a godsend or an unnecessary layer to get through?

Neither, really.

Layering is necessary because without it, you reach a point where your system becomes unmaintainable spaghetti. It's also one of the tenets of reusability: if the developer of a library did a good job, the people using it shouldn't have to care about the details of the implementation. The amount of canned code we use in our systems has grown by orders of mangitude from what it was when I wrote my first program 35 years ago. That growth means we're able to do more powerful things as time goes on. This is good.

The place where it's been a problem for me is entirely cultural. My pragmatic half understands that it's no longer possible to wrap my mind around every last detail and be able to finish the things I want to get done. (Getting older doesn't help, either.) My cantankerous graybeard half had a difficult time letting go of many years of having such a fine-grained understanding of everything I worked on.

Do you think younger programmers would benefit more learning low-level programming BEFORE exploring the realms of expansive libraries?

As has been pointed out in other answers, there's a balance to be struck between attracting and maintaining the attention of neophytes and giving them an ideal, from-the-bottom-up education. If you can't do the former, the latter can't happen.

I see things in our industry that parallel the rest of society. It used to be that almost everybody grew their own food and spent a lot of time doing that. Since then, we've sprouted specialists called farmers who do that job, freeing up others to do other things that contribute to society. I buy my food at a grocery store and would be thoroughly unable to produce most of it on my own if I had to. We have a similar thing going on, albeit on a much more compressed time scale. Programmers are specializing in some set of layers and not others. The average guy writing GUIs may know that there is such a thing as swap space but probably doesn't know or care much about how the operating system is managing it.

The upshot of all this is that it's no longer about just development. Continued specialization means developers will need to continue improving their communication and integration skills.

share|improve this answer
1  
Would the downvoter care to expound on how I could improve this answer? –  Blrfl Jan 15 at 21:24

As with everything, a little bit does you good, but too much hurts. The problem is too many systems don't know when to stop - just 1 more abstraction, to help you program faster... but then you end up coding in the real world where things are never quite as simple as you want, and you spend more time working round the edges than you would have spent with a less featured abstraction.

Its ably described here

or here - "with a single line of code you could add 500 users to the domain"...

Your abstractions try to hide the complexity from you, but really all they do is hide that complexity. The complexity is still there, its just that you have much less control over it - and that's why you end up with this kind of situation.

share|improve this answer

Do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?

I don't think so. There are still plenty of situations in where it's beneficial to be aware of low the 'layer below' works, e.g.

  • When debugging a problem on layer n, it's can often be explained by considering what happens on layer n-1 (i.e. the layer below). I guess layer 0 would be "transistors" but if you want to explain a problem with transistors you would probably talk about physics (e.g. heat), so maybe physics is really level 0.

  • When optimizing code it (unfortunately) does help at times to lower the level of abstraction, i.e. implementing an algorithm in terms of a lower-level layer. However, compilers became really good at doing this for you if they actually see all the code involved. This reason became more popular recently with the boom of mobile and embedded devices, which tend to have weaker processors and where "performance per Watt" is much more relevant than on, say, desktop systems.

In general however, it became a lot easier to make computers do stuff (even if in slightly inefficient ways) which means that there are a lot more programmers than there used to be. This in turn made the "human" factor much more important: Robert Harvey's answer already mentioned that "humans can only hold so much knowledge in their heads at any given moment", and I think that's a very relevant aspect nowadays.

A major motivation in programming language and library (i.e. API) design is to make things easier on the human brain. To this day, everything still gets compiled down to machine code. However, this is not only error-prone, it's also notoriously hard to understand. So it's very desireable to

  • Have the computer help you in finding logical errors in the programs you write. Things like static type systems or source code analyzers (I hear Eric Lippert works on a fairly popular one these days) help with that.

  • Have a language which can be processed efficiently by a compiler and which communicates the intent of the programmer to other programmers to make working on the program easier. As an absurd extreme, imagine writing programs in plain english was possible. Fellow programmers might have an easier time to imagine what's going on but still, the description would be very hard to compiler into machine instructors, and it's notoriously ambiguous. So you need a language which a compiler can understan but which is also comprehensible.

Given that a lot of (most?) compilers are still very much general-purpose, they feature a very generic instruction set. There's no "draw a button" instruction or "play this movie". Hence, moving down the abstraction hierarchy makes you end up with programs which are very hard to comprehend and maintain (though trivial to compile). The only alternative is to move up the hierarchy, leading to more and more abstract languages and libraries.

share|improve this answer

"if older programmers see innovations like these as a godsend or an additional layer to abstract through, and why might they think so?"

I've been programming since I was in high school, about 34 years, starting with Basic and Z80 Assembler, moving to C, various 4GL languages, Scheme, SQL, and now various Web languages. The scope, scale, and depth of the problems addressed by the profession experienced an inflationary period over that time, particularly in the 1990's. Constructs such as libraries, frameworks, and OS services are all contrivances meant to address the complexity that goes along with the expanded space of problems. They are not a godsend nor a burden in and of themselves - just a continued exploration of a vast solution space.

But, IMHO, "innovation" is better understood in terms of novel forms, and not mistaken for sideways movement - reintroducing forms we've already seen introduced. In some ways, the fecundity of an ecosystem suffers when the primitive forms don't compose, when they fixate decisions made early in the evolution, or can't reprocess their own detritus. Some, if not most, of the constructs we remain focused upon do not prioritize long-term sustenance of value as a concern. That has started changing - approaches like Service Orientation and Domain Driven Design, not to mention hypertext and graph-based models, for instance are altering the landscape. Like any ecosystem, eventually the dominant forms will give way to new forms; we are best served by allowing diversity, broadly and in small increments rather than large-scale top-down standardization followed by all-at-once collapse.

"And do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?"

I would argue that most human language is based on metaphors long since forgotten, so while I'd support learning low-level programming from a scientific/numeric literacy standpoint, it is more important that we seek primitives that will support the scale and scope of the problems we are tackling in a way that we can safely ignore the lower level of detail. A framework isn't a primitive, nor is an OS or a library - they are fairly poor at doing the kind of abstraction we really need. Real progress will take people who (a) know what went before and (b) can think in a novel enough manner to come up with something different enough to explore a space of solution that hasn't been explored before or was explored and forgotten.

OTOH, even if your goal is to work as a technician/mechanical level, some level of low-level programming exposure is still going to be helpful to developing your problem solving skills.

share|improve this answer

protected by maple_shaft Jan 15 at 17:04

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.