In such cases, can I somehow use a profiler to determine improvement [...]?
It's very helpful to just repeatedly run the profiler over your code as you are tuning. It also points out not only how the times are changing as you tune but also how the dynamics are changing (reduced branch mispredictions, cache misses).
Also as you go, you can watch how the hotspots kind of reshuffle. For example, after optimizing your top hotspot, it might drop to 8th place and the system puts a focus on a new hotspot (and it might not even be what was shown as the second hotspot initially due to changing dynamics, especially if you are shifting memory layouts for cache-friendliness and exploring alternate data structures).
I find that these kinds of sessions also help me understand the nature of the hardware at an intuitive level just a little teeny bit more each time just by watching the cache misses, branch mispredictions, and the way the hotspots and shuffling with each change. I'd never claim to be able to predict the nature of the hardware though (why I lean so heavily on the profiler) but profiling repeatedly does help me kind of develop some sense of intuition, even more so when my efforts make things slower (learning from mistakes).
Building that intuition helps me design more efficient code the first time around without degrading maintainability, and also helps me understand how to design my interfaces better in a large-scale system in a way such that optimizations can be applied without changing the interfaces (only local implementation details). Understanding how efficiency works at the computer architecture level can help you write cleaner code, since hard-to-maintain code often comes about from dystrophy of repeated iterations resulting from unforeseen changes that degrade maintainability until the code is eventually refactored. Understanding performance a little better upfront can help mitigate that kind of degradation by getting your designs more correct the first time around in places where you know in advance that you'll be dealing with some very tight loops over very large inputs.
What Not to Optimize
For people who are kind of reluctant to use profilers (too proud after getting decent results without one for years, e.g. -- been there, done that *), you might find that you are reasonably successful at spotting the number one hotspot in your system on your own without the aid of such precise measuring tools.
* As an embarrassing personal story, I was one of those stubborn types insistent on not using a profiler, and went for almost 20 years without using one. It's because I was getting pretty decent results without one and knew my way around the code, a kind of classic case of man vs. machine. The thing that changed it and forced me to use one was a conflict scenario where accusations were made among a team about who was causing bottlenecks in the codebase based on superstitions about what language features may or may not be slow. I finally reached for a profiler then to try to settle the matter objectively, only to realize I found a tool that was almost as invaluable as a debugger through the process. It is a natural inclination for a veteran in the industry who has done well without a profiler to look down upon one, but once you embrace one, the whole superstition against them will start to seem really silly. It's not about our ability to write efficient code. It's about our ability to do that more productively.
Yet the greatest immediate value of a profiler, perhaps, is in telling you what not to optimize. Here is an example of a personal profiling session when loading a text-based format (OBJ) format containing 16 million polygons:
Drilling into the #1 hotspot, it shows the times are significantly dominated by string parsing with functions like
sscanf. Without the profiler, I would have never guessed that the mesh engine took so little time. I would have been tempted to at least tweak and tune functions to create polygons and so forth otherwise. My hunches there would have been that the mesh system contributed at least a good 35-50% of the time with the other in string parsing, when the mesh engine is actually only contributing about 12%.
So the profiler immediately gave me a tremendous boost to productivity by showing me that all those parts I suspected would have taken at least a bit more time here don't need to be optimized. That's something that only the accuracy of a profiler can provide (alternative human measurement techniques generally don't come close, even if they can do an okay job at pointing out what to optimize).
If you use a profiler for a good number of years, optimization will cease to seem like some absolute, independent metric. It'll start to seem like a productivity metric. You start to appreciate what the profiler does for you productivity-wise more than just pointing out bottlenecks. When the profiler shows you what doesn't need optimization, you can write such code to be as easy to maintain as possible without a single performance worry.
Testing for Correctness
It's critical to lean on your unit and integration tests before you go too deep, and run them with each optimization attempt, since it's horrible if you find after 3 iterations worth of optimizations that your tests fail after you get excited over the improvements, only to then realize you introduced a glitch and have to back them out (something I've done a few too many times to care to admit).
Here's a recent profiling session I applied with VTune. I made no algorithmic improvements, only micro-optimizations like multithreading, SIMD, and the bulk of the times dropping as a result of memory optimizations (fixed allocator, more contiguous layouts, shuffling data members in
structs to reduce padding, hot/cold splitting, etc). Towards the end there most of my attempts to speed things up were unsuccessful and I had to back out my changes half the time, and the few improvements that were successful were getting increasingly diminished returns (with noise making it difficult to even tell if I made a genuine difference). That's usually a good sign to kind of stop and rest and let the code breathe for a while.
T-Rex (12.3 million facets):
Initial Time: 32.2372797 seconds
Multithreading: 7.4896073 seconds
SIMD: 1.7831 seconds
4-valence patch optimization: 1.25007 seconds
I like to open up notepad when I do this stuff and just record the times from VTune with each step, also with some "checkpoints" recorded where I did something significant (like the SIMD optimization). It helps me keep track of the times and how they relate to each iteration of change.
Also when you go through an optimization crunch like this, it's useful to start a new branch in your version control system and record the time differences in your commit logs as well.
Start Off Big
One thing you want to do is make sure you use a large enough input with big times showing up initially (bigger than you would initially be tempted). Otherwise you can find that after you optimized things 10x, e.g., the times are getting too small to effectively measure without noise messing them all up, and that can tempt you to reach for a bigger input to inflate the times again, but that makes it harder to keep track of the improvements. So initially it's good to make a test that takes like 30 seconds, 60 seconds, even if it's annoying to wait that long, since you might end up optimizing to the point otherwise where the times are getting too small to effectively measure.
This is especially true in desktop apps which run constantly and it is
difficult to: execute the same path and execute it the same number of
times to have reliable comparison.
This might be overkill but I've found it incredibly useful here for very coarse integration testing and also performance testing to be able to write scripts for your application at a high-level, almost macro-like fashion (expressing a user-end action with a line of code). It's how I tend to do profiling sessions -- I run a little lua script which looks like this (pseudocode):
load some file
perform some operation, possibly multiple times
quit application (this also reports the total time the application ran)
I use LuaJIT for the purpose (mainly because its FFI makes it possible to directly call exported C functions from Lua) though that's just a personal preference. With that, it's easy to just write a few lines of script that drive your complex application and run a profiler like VTune over it with command arguments that specify the script to run. Users might also appreciate this ability to externally drive the application and make it perform batch jobs.
If you end up doing this, it's also invaluable for bug reports. If a user reports a bug with a minimal number of steps and possibly content to reproduce an issue, you can start off writing this kind of high-level macro script to reproduce the bug and make the test fail. Then you have the simple task of making the test succeed, and can include it as part of your CI process.