The tag has no wiki summary.

learn more… | top users | synonyms

58
votes
10answers
6k views

Is it premature optimization to add database indices?

A colleague of mine today suggested that we go through all of the queries in our application and to add indices accordingly. I feel this is premature optimisation because our application is not even ...
22
votes
8answers
2k views

Should my (sequential) collection start at index 0 or index 1?

I am creating an object model for a device that has multiple channels. The nouns used between the client and I are Channel and ChannelSet. ("Set" isn't semantically accurate, because it's ordered ...
17
votes
1answer
13k views

What is the difference between btree and rtree indexing?

I've noticed on MySQLWorkbench that you can choose how to store your indexes before forward engineering your design. The storage types are: BTREE RTREE HASH Researching this, I found some ...
12
votes
6answers
2k views

Do You Still Need Indexing After Database Normalization

After you have done a good normalization, do you still need to index the table? How is this going to affect the performance? Will it even affect the performance in some way after a good normalization? ...
12
votes
5answers
1k views

How come the computer doesn't have to read the entire table when the column is indexed? [duplicate]

Let's say a table with two columns has 100 quadrillion records. And I want to find a record that has column #2 equal something. If column #2 is indexed it returns the result immediately, but if it's ...
12
votes
6answers
25k views

How to quickly search through a very large list of strings / records on a database

I have the following problem: I have a database containing more than 2 million records. Each record has a string field X and I want to display a list of records for which field X contains a certain ...
8
votes
1answer
444 views

How do I efficiently store all OpenStreetMap data in an indexed way?

I have a PBF file that contains the following information about a country: Nodes, each with their own longitude, latitude and properties; used to store points in a 2D space. Ways, each with their ...
8
votes
1answer
380 views

lightweight document indexing to handle less than 250k potential records

Recently I've found myself chafing at the limitations of document indexing engines. I was developing a small website that needed some fairly robust searching capabilities but due to their hardware ...
5
votes
4answers
550 views

Randomly Accessing Data Packets in a Compressed File

In my line of work I deal with very large files, hundreds of Gigabytes in size. The nature of the data in these files is such that compression would greatly reduce their size. The problem is, the ...
4
votes
2answers
524 views

Fast set indexing data structure for superset retrieval

I am given a set of sets: {{a,b}, {a,b,c}, {a,c}, {a,c,f}} I would like to have a data structure to index those sets such that the following "lookup" is executed fast: find all supersets of a given ...
4
votes
4answers
265 views

Quadtree with duplicates

I'm implementing a quadtree. For those who don't know this data strucutre, I am including the following small description: A Quadtree is a data structure and is in the Euclidean plane what an ...
3
votes
3answers
1k views

storing and retrieving millions of documents using c#

I am working on an integration project, where my “app/web service” will sit in the middle serving documents. Basically, a request is sent with the document id as part of the query string, I check if ...
3
votes
3answers
324 views

Real world analogy for a clustered index

A db index is analogous to a table of contents. This helps me understand db index in an easy way. My question is are tehre any real world analogies for a clustered index?
3
votes
1answer
4k views

Database design for tagging system

I'm creating a system where items will have one or more tags. Now there will be an item table that will store each item, and a tags table, that will store all tags used in the system. My question ...
3
votes
2answers
287 views

What is a good stopword in full text indexing?

When you go to the Appendix D in Oracle Text Reference they provide lists of stopwords used by Oracle Text when indexing table contents. When I see the English list, nothing puzzles me. But the ...
2
votes
6answers
4k views

How many indexes is too many indexes?

I have a table with about 37 indexes. Would that be considered too many? What are some best practices regarding indexing? EDIT: It is warehousing data, updated daily with new transactions. There are ...
2
votes
4answers
980 views

Algorithm for indexing strings in “list”

Imagine I have file called strings.dat. Inside this file there is alot of string, for example: one million. String are sorted. Now I want to find specified string, so I can write method like this: ...
2
votes
2answers
83 views

Efficient range search for pair of numbers

Assume we have large list of pairs: struct {x: double, y: double} pair; vector<pair> What is the most effective way to find all pairs where (x1 < x < x2) AND (y1 < y < y2)? O(n) ...
2
votes
1answer
477 views

Which is the best method for storing data for a Language Dictionary?

I have been working on making a completely offline dictionary using the Wiktionary XML dumps. The dumps themselves are about 10 MB, but when converted into a index using a search engine indexer (I use ...
2
votes
1answer
140 views

Finding the closest n points to any arbitrary point in two dimensions (r-tree, quadtree, spatial index)

I have a distribution of two dimensional point objects. How is it possible to find the nearest N number of points to any given point without iterating over the entire collection of points (and only ...
1
vote
1answer
755 views

What exactly is a Search Engine Indexer. Where to start building one?

For one of the projects at uni we were given the task to create a custom, niche search engine. My colleges and I split the tasks amongst each other so that we can tackle the overall project more ...
1
vote
2answers
334 views

Data retrieval and indexing

I have around 800,000 rows of data stored in the boost shared memory from the database. The data are in the form: Id Color Length Size 1 1 2 ...
1
vote
1answer
55 views

Difficulty in deciding correct data structure

Two objects are interacting (Object Alpha, Object Beta) Each contain a point (x,y) which will be used to make comparisons, among other things. Object Alpha's point (x,y) attribute is dynamic and ...
1
vote
4answers
180 views

Ideas for scraping a home-brewed Php-powered bug system [closed]

I work at a large company, and we are stuck with a crappy home-baked bug tracking tool, which does too many things half-decently, but it search functionality is next-to-impossible to use (has about 80 ...
0
votes
3answers
1k views

How much better is Apache Solr than its non-Java competition? [closed]

How much better is Apache Solr than its non-Java competition? Is it so much better than other solutions, e.g. this one (Sphinx with PHP API) or Zend_Search_Lucene that it would be worth suggesting ...
0
votes
1answer
735 views

Wildcard user in select statement on indexed varcharcolumn

This question is based on some testing I've done recently in order to better understand SQL indexing system. Table with 500k entries, InnoDB engine This is a simple select query on a varchar field. ...
0
votes
2answers
167 views

Dissection of Google indexing services

There are more than a few questions that hop into mind when someone thinks about Google's indexing services. Jeff Atwood wrote about them at The Elephant in the Room: Google Monoculture and Trouble In ...
0
votes
1answer
249 views

Full Text Indexing Strategy for MS Excel Documents

Background As part of a broader application that allows users to search thousands of MS Office documents on a private network, I need to index and make searchable Microsoft Excel files. My basic ...
-1
votes
1answer
47 views

node deep file indexer module cannot go deep [closed]

I'm trying to make a file indexer by node.js. The program is supposed to index files and folders in an array and also check inside folders and add all sub folders and files I wrote: ...
-2
votes
2answers
228 views

Using database Indexes [closed]

This might sound like a naive question but when are database table indexes in mysql required? How do such indexes affect performance and why?