New answers tagged

1

The most efficient would probably be a binary format which you could read directly into memory, skipping the parsing step.


2

You can do better than either XML or JSON by exploiting the particular characteristics of your particular data. If the data is completely flat and every row contains the same fields, then CSV is going to be more efficient than either XML or JSON. The reason people generally prefer formats like XML and JSON is because they recognise that the life-time cost of ...


4

Running your queries in parallel from your application, and then combining the output will never be as fast as running a tuned query from the database. Do the work where it is most appropriate Database engines are designed and optimized to work with data sets. Certainly some database engines are more efficient at this than others, but this is their primary ...


0

The ANSI SQL standard does not require hash indexes. Several relational database products have chosen hashing as an index implemention, however. PostgreSQL allows them to be explicitly declared. SQL Server uses hashing in its in-memory OLTP engine. Also, one way of implementing a SQL JOIN condition is to use hash tables, though these hash tables live only as ...


0

I believe that using a single SELECT with a more complex WHERE clause is a better alternative, in terms of design. I would also expect that, somewhat by chance, the UNION approach is slower because the database needs to eliminate duplicates. If you were using UNION ALL instead, I wouldn't be able to guess which alternative would have better performance ...


0

Performance wise you will get the parallel run better but code wise it might be better to put everything in the stored procedure in a union. In future if you need a modification ( say an update of the data fetched by joining some other table ) or say an other 3-4 unions added to the query . Now you will have to refactor your code which calls the stored ...


0

Since all your data comes from the same table this is an odd use of union. Instead you could do a single select with a more complicated where statement. Now! whether one is more performant than the other I think will depend on the where clause. I'm not sure how you mean to run them in parralell? call them from code and iterate the results? This will be ...


3

Hash tables allow you to very quickly lookup an item if you have the exact key. Hash tables cannot handle a request like "give me all invoices issued from the 2nd to the 15th of April". A lot of a time a database access wants all data in some range of values. Search trees work a lot better for that.


1

All you can really do is profile—any general advice is going to be very, well, general. You should try to keep your working set small so that it fits in the first levels of cache, and avoid redundant memory accesses. If it’s expensive to compute an intermediate value, precompute it and store the result. If you know you will need data, prefetch it from RAM. ...


1

What I would do. Repackage the data on disk in small cubes, a few KB each. That is, if the data are represented as a typical array of arrays of arrays, the points that are close in the 3D space are not close in the representation. I'd try to overcome that. The result would be a the same amount of data, addressable in two steps. First step would be finding ...


-1

Code performance isn't the only argument for changing code. Is the code easier to understand which the changes you propose? Do those changes make the code easier to maintain? If the performance of a piece of code isn't critical than maintainability and being free of bugs is often more important.


1

You can either do things the wrong way, or do things the right way. Often, things are done the wrong way and code is refactored so that it's done the right way. If you're going to write new code, and you know that you can do things the right way without a major penalty, I'd just err on the side of doing it the right way. (Note that, after performance ...


2

In my experience, if you get this kind of opposition to optimization regularly, people are not really complaining about optimization. They are complaining about what you are sacrificing in the name of optimization. This is usually readability, maintainability, or timeliness. If your code is delivered in the same amount of time, and just as easy to ...


4

I think the full quote in context is instructive. I'll copy from a post I made on Reddit on the topic: There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong ...


2

There is indeed lots of misunderstandings about this quote, so it's best to step back and look at what the actual issue is. The issue isn't so much that you should never "optimize". It's that "optimize" is never a task you should set out to do. You should never wake up in the morning and say to yourself "hey, I should optimize the code today!". This ...


32

It seems you are looking for shortcuts not to try out the "purest naive implementation" first, and directly implement a "more sophisticated solution because you know beforehand that the naive implementation will not do it". Unfortunately, this will seldom work — when you do not have hard facts or technical arguments to prove that the naive implementation is ...


8

How to work with people who stonewall a discussion the minute it has to do with performance? Begin with shared principles that build on the strategic direction of your group. My Principles: My personal principles on writing code are to first aim for correctness in my program, then to profile it and determine if it needs optimization. I profile my code ...


17

Ask yourself this: Is the software NOT meeting performance specification? Does the software HAVE a performance issue? These are reasons to optimize. So, if people are opposed, just show them the specification and go back to them and explain we need to optimize because we are not meeting spec. Other than that, one would have a hard time convincing ...


-2

Optimisation is evil. Use performance tuning. Performance tuning makes everyone happy. Note that performance tuning isn't aimless like "optimisation", it very deliberately looks for aspects of your application that objectively don't perform well enough and improves them with a measurable outcome. You may also remove bugs that lead to slowdowns. Which ...


6

The way forward is to forget about the actual quote and the various interpretations - it it dogmatism either way to focus so much on a specific quote by a guru. Who says Knuth is always right anyway? Instead focus on the project at the hand, the piece of software you are developing along with the colleagues you disagree with. What is the requirements for ...


1

This seems like a communication problem and not a programming problem. Try to understand why people feel the way they do and try to crystallize why you think your way would be better. When you've done that, don't start a confrontational argument where your goal is to tell others why they're wrong and you're right. Explain your thoughts and feelings and just ...



Top 50 recent answers are included