Questions regarding efficient storing and representing data within a software application.

learn more… | top users | synonyms

1
vote
1answer
176 views

Storage and naming of log data

I'm in the process of rewriting a small application from console mode to a GUI mode. The input is a file with log data from a measuring device: ... 00029;00044;00076;00044;00021 ...
3
votes
1answer
290 views

How do I transparently cache data in intermediate stages of processing?

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline. ckt = generate_ckt(ckt_properties); freq = generate_fpoints(fconfig); result = ...
2
votes
1answer
134 views

Applications with duplicate data - keep separate or combine into one set?

We currently have 3 similar, but slightly different applications that use the same data. We load the (same) data into each application that uses it. The applications are similar and use the same ...
6
votes
3answers
3k views

What is the best way to store a table in C++

I'm programming a decision tree in C++ using a slightly modified version of the C4.5 algorithm. Each node represents an attribute or a column of your data set and it has a children per possible value ...
0
votes
1answer
130 views

overworld vs on-screen 2d tile organization

I am developing a 2d game where the world is made up of tiles. I have an overworld and a "current world": the current world represent the current tiles on screen at any given time, currently this ...
1
vote
1answer
302 views

Database data structures for RESTful api

I am creating a RESTful API. I am struggling to decide on the best way to design my database tables around my resources. Initially, I though a table per resource would be a good way to go, but I'm ...
2
votes
2answers
2k views

Nested Enum type in C++ or C#?

I've come across a recurring issue in a few of my recent projects in which I find myself using enums to represent state, or type, or something else, and I need to check against a few conditions. Some ...
2
votes
1answer
288 views

Marked nodes in Fibonacci heaps

I don't understand why Fibonacci heaps have marked nodes (picture). A node is marked when its child is deleted. Quoting from Wikipedia: "[Keeping degree of each node low] is achieved by the rule that ...
2
votes
1answer
106 views

creating simple states for a stateless input

Given an input device (basically a keyboard) that reports keyup and keydown, how may I most efficiently store and retrieve information about which keys are currently depressed? My first thought was a ...
0
votes
2answers
222 views

Data structure for fast StartsWith filtering of a dictionary

I have a list of english words. I'm trying to come up with a data structure that allows me to do fast StartsWith filtering, like: var words = dictionary.StartsWith("foo"); I was thinking of ...
18
votes
2answers
2k views

What's the difference in content between Chris Okasaki's 1996 thesis and 1999 book, Purely Functional Data Structures?

I want to read Purely Functional Data Structures. I've easily found the thesis (which is freely available as a PDF), but see that there's a book available also. So I'd like to know what the ...
11
votes
7answers
13k views

What is the point of using lists over vectors, in C++?

I've run 3 different experiments involving C++ lists and vectors. Those with vectors proved more efficient, even when a lot of insertions in the middle were involved. Hence the question: in which ...
7
votes
2answers
2k views

Using a stream manipulator (endl) or a newline escape character (\n)?

I don't have a specific context in which I'm asking the question, but while I was reading a beginner book on C++ I noticed the use of both an endl stream manipulator and a newline escape character ...
0
votes
2answers
531 views

Algorithms for data structures in distributed system

The hash table data structure can be easily spread across multiple machines with a simple algorithm to distribute the keys: machine_to_query = item_key % machine_count When you want to read and ...
5
votes
2answers
916 views

How do scalable bloom filters work?

I was reading up on scalable bloom filters and could not understand how each time a constituent bloom filters fills up, a new bloom filter with larger size is added. The elements that contributed to ...
2
votes
2answers
6k views

How to increase the efficiency of an Immutable Queue in java?

I have been developing an Immutable queue in java, but the version I have developed is supposedly slow and I have been suggested to create a faster version of the same and I have no idea as to how to ...
2
votes
2answers
866 views

Thoughts on a custom data structure for a node/chain implementation

I am designing a simple GUI that will allow a user to create a GPX route by clicking repeatedly on a map panel. I am faced with a design dilemma for how to represent the nodes and route. This ...
4
votes
2answers
980 views

Creating huge decision tree

I'm to write an event correlator. A fundamental part of the system will be a decision tree that recognizes the origin of the fault basing on recorded states and log files. Often many accidents will ...
3
votes
6answers
530 views

Should a database table(s) structure match its intended data structure(s) in the logic?

This question branches out of this question, What are the differences between algorithms using data structures and algorithms using databases?. The General Question Should a database table(s) ...
6
votes
4answers
457 views

Representing a rule in a ruleset

How to represent rules for a rule engine as objects? A rule would be if (booleanExpression(input)) then a chain of generic actions" else next rule ...where the generic actions might be e.g. ...
7
votes
4answers
2k views

What are the differences between algorithms using data structures and algorithms using databases?

The General Question What are the differences between algorithms using data structures and algorithms using databases? Some Context This is a question that has been bugging me for some time, and I ...
1
vote
3answers
876 views

What data structure could a word processor use to map the user's caret position/text selection to its internal representation of the document?

Word processors (e.g. Microsoft Word) display documents to the user as styled text. The user can select a part of this text and apply styles to it, and edit the text. The word processor must (I ...
3
votes
1answer
119 views

Representing domain objects

This is related to my recent question regarding naming awkward domain objects. A number of answers indicated I was using the wrong representation for the domain objects. To summarize, I chose to use ...
-1
votes
1answer
753 views

Simple dependency tree diagram generator [closed]

I have a need to produce a simple dependency tree diagram. The input data would be in the following simple format: ITEM_NAME DEPENDENCY ---------------------------- ITEM_101 ITEM_75 ITEM_102 ...
3
votes
2answers
3k views

How, when, and why do developers use custom data structures in python?

Python's lack of pointers makes data-structure construction intuitively more challenging, and has so much built in functionality that, as an amateur, I can't see when, why, or how you would create ...
20
votes
3answers
772 views

How do I express subtle relationships in my data?

"A" is related to "B" and "C". How do I show that "B" and "C" might, by this context, be related as well? Example: Here are a few headlines about a recent Broadway play: David Mamet's Glengarry ...
4
votes
3answers
766 views

Is it conceivable to have millions of lists of data in memory in Python?

I have over the last 30 days been developing a Python application that utilizes a MySQL database of information (specifically about Norwegian addresses) to perform address validation and correction. ...
16
votes
1answer
11k views

Sets Data Structure in Golang

I really like google golang but could some one explain what the rationale is for the implementors having left out a basic data structure such as sets from the standard library?
-3
votes
1answer
656 views

map data structure in pacman [closed]

I am trying to make a pacman game in c#. I have done some basic work and have previously replicated games like copter-it and minesweeper. I am confused about how to implement the map in pacman. Which ...
5
votes
3answers
355 views

Generating Deep Arrays: Shallow to Deep, Deep to Shallow or Bad idea?

I'm working on an array structure that will be used as the data source for a report template in a web app. The data comes from relatively complex SQL queries that return one or many rows as one ...
4
votes
2answers
951 views

Which design better when use foreign key instead of a string to store a list of id

I'm building online examination system. I have designed to table, Question and GeneralExam. The table GeneralExam contains info about the exam like name, description, duration,... Now I would like to ...
4
votes
2answers
459 views

Fast set indexing data structure for superset retrieval

I am given a set of sets: {{a,b}, {a,b,c}, {a,c}, {a,c,f}} I would like to have a data structure to index those sets such that the following "lookup" is executed fast: find all supersets of a given ...
5
votes
2answers
1k views

Data structure for pattern matching

Let's say you have an input file with many entries like these: date, ticker, open, high, low, close, <and some other values> And you want to execute a pattern matching routine on the ...
0
votes
3answers
195 views

I need some help creating a non-binary tree (or some other data structure that will better solve my problem)

I have about ten lists of numbers and some strings. Each list has about <= 30K lines. Each line on a list has a distinct number. I need to build an efficient way of finding all the lines in each ...
4
votes
2answers
863 views

Patterns for a tree of persistent data with multiple storage options?

I have a real-world problem which I'll try to abstract into an illustrative example. So imagine I have data objects in a tree, where parent objects can access children, and children can access ...
31
votes
7answers
8k views

When to use DAG (Directed Acyclic Graph) in programming?

I recently found a framework named ecto. In this framework, a basic component named "plasm", which is the ecto Directed Acyclic Graph.In ecto, plasm can be operated by ecto scheduler. I am ...
6
votes
1answer
291 views

How to design data structures for sharding?

When designing data structures for an application, what are some best practices to make sure it can be sharded? In particular, which of the following is better: Creating relationships in databases ...
6
votes
1answer
2k views

What is the usage of Splay Trees in the real world?

I decided to learn about balanced search trees, so I picked 2-3-4 and splay trees. What are the examples of splay trees usage in the real world? In this Cornell: ...
3
votes
2answers
488 views

I have data that sends in “bursts” of 100 records with a significant delay. How do I structure my classes for multithreading?

My datasource sends information in 100 batches of 100 records with a delay of 1 to 3 seconds between batches. I would like to start processing data as soon as it's received, but I'm not sure how to ...
3
votes
1answer
994 views

How should I architect my Model and Data Access layer objects in my website?

I've been tasked with designing Data layer for a website at work, and I am very interested in architecture of code for the best flexibility, maintainability and readability. I am generally acutely ...
1
vote
2answers
649 views

Count function on tree structure (non-binary)

I am implementing a tree Data structure in c# based (largely on Dan Vanderboom's Generic implementation). I am now considering approach on handling a Count property which Dan does not implement. The ...
1
vote
6answers
5k views

Serializing Data Structures in C [closed]

I've recently read three separate books on algorithms and data structures, tcp/ip socket programming, and programming with memory. The book about memory briefly discussed the topic of serializing data ...
5
votes
2answers
264 views

Data structures for storing finger/stylus movements in drawing application?

I have a general question about creating a drawing application, the language could be C++ or ObjectiveC with OpenGL. I would like to hear what are the best methods and practices for storing strokes ...
2
votes
1answer
279 views

2D grid with multiple types of objects

This is my first post here in programmers.stackexchange (I'm a regular on SO). I hope this isn't too general. I'm trying a simple project to learn Java from something I've seen done in the past. ...
2
votes
1answer
160 views

Separating validation from persistence in models?

Models in an MVC application gets kind of hard to test when validation of data and persistence of data is baked together in the model. I would like to test those separately to prevent doing ...
3
votes
3answers
498 views

Internal Mutation of Persistent Data Structures

To clarify, when I mean use the terms persistent and immutable on a data structure, I mean that: The state of the data structure remains unchanged for its lifetime. It always holds the same data, ...
6
votes
1answer
162 views

Unit testing a text index

Consider a text index such as a suffix tree or a suffix array supporting Count queries (number of occurrences of a pattern) and Locate queries (the positions of all the occurrences of a pattern) over ...
5
votes
2answers
804 views

Efficient Data Structure for holding a graph

Problem Link - http://opc.iarcs.org.in/index.php/problems/NUMTRIPLE In my opinion, the problem can be solved by a data structure, that shows how each number is connected to another, and via recursion ...
3
votes
1answer
474 views

Amazon Kindle - Whispersync implementation?

For those who are not aware of Kindle's whispersync, here is how it works (from amazon.com): "...Whispersync synchronizes the bookmarks and furthest page read among devices registered to the same ...
0
votes
1answer
55 views

Create association between informations

I deployed a project some days ago that allow to extract some medical articles using the results of a questionnaire completed by a user. For instance, if I reply on questionnaire I'm affected by ...