Micro-Optimizations Aren't Micro-Impact
I'm reminded recently of a comparison I made between a BVH implemented for an open source raytracer vs. Embree (a raytracing kernel made by Intel).
Both used the exact same data structure, a bounding volume hierarchy, so same exact algorithm/data structure, and both offering logarithmic O(logN) tree traversal to find a triangle that intersects a ray.
Given the same scene:
- Embree, micro-optimized by some of the industry's finest, managed to process 112 million rays per second.
- The BVH I was comparing this to (which only seemed to be implemented to fulfill algorithmic requirements, little or no focus on micro-optimization) could process 310k rays per second.
So we have Intel's version which is
~361 times faster simply as a result of micro-optimizations.
Micro-optimizations, applied correctly by people who understand their fair share of computer architecture, the nature of the optimizing compilers they use, and how to measure hotspots, don't necessarily have a microcosmic impact. They can, when applied correctly in the right circumstances, easily make things dozens to hundreds and occasionally even thousands of times faster.
We Live on Micro-Optimizations
We live and breathe and thrive on micro-optimizations, whether we know it or not. The very operating systems we use have been micro-tuned to death (imagine the Linux kernel being implemented in a very high-level language by developers who constantly scold those with a predominant focus on computer architecture and compiler optimizations).
Whenever we scroll web pages with a mouse wheel, we're relying on micro-optimized image rasterization functions that have been written at the bare metal bits and bytes level to get an interactive response while doing millions of iterations worth of pixel processing work. It is only through micro-optimizations that our machines can process hundreds of millions of pixels per second. Lacking them, the times can quickly degrade to a mere million pixels per second, e.g., at which point we might have to wait a full second just to see a response when trying to scroll a webpage.
Even the compilers we use were built by people with a predominant focus on micro-efficiency at the register and instruction level, yet that obsession is far from premature.
Whenever we type something to search for against a popular search engine, the backend of that search engine will be micro-tuned to death to save costs on an already-enormous electricity bill.
Innovation Requires Micro-Optimization
By "innovation" here, I merely mean functionality that is not covered by an existing library, framework, operating system, compiler, programming language, etc. In effect, one where we actually have to create a new solution largely from scratch. People can be very creative and "innovative" in the way they assemble existing solutions out there written by other people, but I'm using "innovative" in a very specific sense and context here.
In those cases, we need micro-optimizations. One of the reasons AAA game developers are often very obsessed with micro-optimizations like SIMD, memory optimization, multithreading, etc. is that they can't just lean on an existing library to do the stuff they do -- it's too "innovative". It's only recently, for example, that we started to see real-time area lights with soft shadows inside a game. Yet even those solutions are far from general, they're very tied to the representation and other design decisions of the proprietary game engine.
Conventional Image Processing Example
For an "uninnovative" (conventional) example, let's consider alpha blending two images together. In that case, most people would be wise to reach for an existing library which has been implemented already in native code and micro-optimized to death. Then our conventional code can merely issue a few very high-level requests to load a couple of images and blend them together.
Yet there is an enormous amount of actual micro-optimized code here. But it's inside the native image library where the library has to actually loop through each and every pixel of both images and blend each pair of pixels together (though very likely with vectorized code to do this in parallel and possibly across multiple threads or GPU kernels).
The conventional code author doesn't have to worry about this since the image library is doing all the performance-critical work for him. The code he writes would just issue these high-level requests, no tight loops involved, easy.. and such an author wouldn't benefit in the slightest by trying to optimize these high-level requests, he might as well process the instructions through a very slow interpreter and it still wouldn't make a noticeable difference.
Yet the author of the image library definitely had to care a whole lot about micro-efficiency, as image processing is all about micro-efficiency. There typically aren't algorithmic improvements in the realm of image processing, since image processing often revolves around this basic linear-complexity algorithm:
for each pixel:
do something with pixel
There's no work to skip here, we have to touch every pixel, and the only way to make this go faster is to use faster instructions, apply better utilization of the CPU cache, SIMD registers, etc.
For someone whose job largely consists of image processing, 75% of their code may actually need, above all else, micro-optimizations, and "micro" here is no longer really "micro", it's just "optimization", as 75% of their code may resemble those tight pixel-processing loops like the above for every single image/video filter they implement.
Unconventional ("Innovative") Image Processing Example
Now let's imagine a case where we're wanting to implement a very unique image processing algorithm which inputs photos of people and outputs an anime-style caricature of them. In that case, we can't necessarily just assemble the desired result out of existing solutions out there. We have to "innovate", and that's when we're almost certainly going to require micro-optimizations if we want to process these images without users waiting ages for a response.
If yes, as hardware improves should we expect higher level languages
to take-over the gaming industry?
Nope, I would say not from the engine "builder" perspective (maybe so for engine "user"). Yet there are probably going to be a fair share of game designers who use high-level languages. There are plenty of game developers who don't need to do anything "innovative" (in the "unconventional" sense I described above) and can just write their game using Unity or Unreal Engine 4, e.g. They don't actually have to make their own AAA engine which pushes new boundaries. Those types just want to focus on high-level game design and logic without getting knee-deep in CPU and GPU processing.
Yet I can't imagine hardware getting fast enough any time soon to the point where it sufficiently meets demands, which also scale with the hardware.
If hardware gets a hundred times faster, for example, suddenly game engines will want to be created which simulate a billion light-emitting particles in real-time. People will start wanting to actually animate characters with muscle rigs in real-time. Beyond that, people will start wanting to model skin pores into their creatures. Enemy A.I. might actually want to be processed through a neural network. All of these kinds of things will always be greedy for faster code, faster hardware, or both.
The demands will continue to scale until we are effectively gods giving birth to life on the machine. I can't foresee when that will happen, if ever, and the human race may cease to exist quickly after that point. Until then, the demands will always be higher than the current trends will provide, and that will push software developers to
optimize their code and hardware developers to try to produce more efficient hardware.
Note that I used "optimize" in the above sentence. Micro or algorithmic makes no difference here, and it often doesn't make a difference when applied by people who know what they are doing. If we had machines so fast that micro-optimizations wouldn't matter, algorithmic optimizations also wouldn't matter. We could just as well bubble sort that million-entry input in such a case.
The goal is speed, and as long as speed is a goal, the need for optimizations are here to stay.
One Possibility: Optimizers
One possible end I forgot to mention is the evolution of optimizing compilers to the point where high-level code starts to increasingly rival the performance of the most expertly-tuned micro-optimizations.
One of the difficulties is that there's always this cat and mouse game between compiler designer, software designer, and hardware designer.
For example, the hardware designer implements branch prediction with the thought that most computer programs will branch in predictable patterns and make all sorts of computer programs faster automagically. Cool, it initially serves its goal. Except soon after, software designers are using profilers which measure branch mispredictions and optimizing their code for branch prediction and getting even faster than average results that way. Meanwhile, the compiler designer is often left in the dark for all but simple, symmetrical, loop-style branches, since they often can't know what common vs. rare case is (requires runtime information and also ideally information about how users will most often use the software). This runtime information barrier gives rise to things like trace JITs which excel when a program is frequently executing the same instructions, but can suffer tremendously if they are not and branching all over the place.
The hardware designer implements CPU caching to speed up memory access with the thought that most memory accesses will be to contiguous memory blocks (array-like in nature). Cool, this initially speeds up even average software. Except now the software designer is running profilers and measuring cache misses and starting to use more contiguous data structures and memory allocators in response. Compiler designers are still ineffective here to even an intermediate-level software developer who understands memory optimizations and knows how to profile things like page faults and cache misses.
Way back in history, compilers were bad at register allocation, so software designers writing in assembly code could easily write more efficient code that resulted in fewer stack spills. At this point the compiler was in a race against man, and eventually compilers started doing an unbelievably good job with register allocation and beating most humans. This was one of the times where the compiler designer actually started leading the race. Except human assembly code writers can still sometimes beat optimizing compilers at instruction selection (requires a tremendous amount of esoteric expertise, however).
Then hardware designers introduced these wide SIMD registers that could vectorize scalar operations and perform them in parallel. Now compilers are just starting to emit efficient SIMD instructions, but the software designers with a sufficient amount of expertise can still often easily beat the optimizer (take it from Intel: they release one of the most aggressive optimizing compilers, ICC, but still write most of their code using handwritten SIMD intrinsics and occasionally even assembly). So now the compiler designers are still racing against the human software designers again, with the humans who have sufficient expertise taking the lead.
It's always this kind of never-ending race. On top of this we have GPUs now which are capable of blazing-fast parallel, symmetrical computation over homogeneous data, and the compilers and tools involved to really take advantage of that are all over the place and about as remote from the idea of "high-level" as we can possibly get.
So there's this never-ending kind of game where software designer comes in the lead, then compiler designer, then hardware designer changes all the rules and kind of resets the game, and it stretches my imagination too much to see it ever coming to an end any time soon.