Questions regarding efficient storing and representing data within a software application.

learn more… | top users | synonyms

9
votes
2answers
592 views

In-memory datastore in Haskell

I want to implement an in-memory datastore for a web service in Haskell. I want to run transactions in the STM monad. When I google hash table steam Haskell I only get this: Data. BTree. HashTable. ...
0
votes
1answer
343 views

Comparing performance of different C++ self-made tree data structures

I have two tree data-structures that I made in C++ , and they give a response for range queries/point queries in that program. I am using Ubuntu. I need help in understanding the way to compare the ...
9
votes
2answers
808 views

Uses of persistent data structures in non-functional languages

Languages that are purely functional or near-purely functional benefit from persistent data structures because they are immutable and fit well with the stateless style of functional programming. But ...
-1
votes
2answers
2k views

What do I need to know about Data Structures and Algorithms in the “real” world [closed]

I just finished the data structures and algorithms course in school , I took it during the summer so 6wks course vs a 16 wk course during the regular semester. So not only was the course hard but it ...
-1
votes
2answers
79 views

Store a tree with near next neighbours

I would to find an efficient structure (speed and volume) to store nodes and their neighbourhood. My input is build out of stings in the following format ./X/Y.log where X ∈ [0,359] and Y ...
0
votes
2answers
406 views

Separation of Concerns, Data Access Layer

I was thinking about this earlier today and figured I would get some input on the matter. When I develop applications I would usually have the Data Access Layer on another project, incase it could be ...
5
votes
2answers
2k views

Optimal way to store 18 billion key, value pairs [closed]

I have around 200 million new objects coming in, and a 90 day retention policy, so that leaves me with 18 billion records to be stored in the form of key-value pairs. Key and value both will be a ...
2
votes
2answers
768 views

Same constructor signature with different semantics?

Occasionally, I'm confronted with the problem that I have to provide some constructors in order to initialize objects with different sets of data. These sets can be mathematically transformed into ...
3
votes
3answers
87 views

How should I manage short-term user preferences, to simplify data cleanup?

When rolling out a new feature, sometimes it makes sense to notify the end user what it is, and give them an opportunity to provide feedback whether the software should get them to 'try something new' ...
5
votes
1answer
678 views

Where to learn graph theory applications [closed]

I am completely new to designing algorithms with graphs. I am following CLRS and other video lectures in Youtube, notably from IIT/MIT. They are pretty good, and I currently have decent idea about ...
1
vote
2answers
929 views

Radix exchange sort implementation?

I need a little help understanding the implementation of radix exchange sort algorithm. Radix exchange (I don't know if it's the exact name in english based literature) is a lot similar to quicksort, ...
4
votes
1answer
2k views

Homogeneous vs. heterogeneous AST representation

What are the reasons to choose a homogeneous vs. a heterogeneous AST representation for implementing a complex domain-specific programming language? Just to be very clear about what I'm asking, here ...
3
votes
4answers
473 views

What's the difference between a stream and a queue?

What's the difference between a stream and a queue? They both have the concept of an ordered set of elements, but tend to have different implementations and a different vocabulary of ...
5
votes
1answer
1k views

Building a Graph Editor - How to create a data driven graph

I am developing a graph-editor that uses drag and drop to build hierarchical graphs (containing nodes and links). Each node in the graph should be linked to a table in our database (SQL Server). I did ...
1
vote
1answer
2k views

How to model/structure data that contains parent/child relationships for the same entity?

I think it is just my thinking that is off here. Suppose I have two entities: question and choice. A question can be a parent or child, or both (in relation to other questions). It seems that this ...
1
vote
0answers
199 views

Single write, multiple read of stateful objects

Scenario: One data input feed (call this Source) Multiple objects (call these Layer1) read this Source, and analyze the feed. This analysis is stateful. Layer1 objects have configuration parameters. ...
10
votes
8answers
3k views

Data structure for accessing units of measure

TL;DR - I'm trying to design an optimal data structure to define units within a unit of measure. A Unit of measure is essentially a value (or quantity) associated with a unit. SI Units have seven ...
2
votes
6answers
3k views

Heap overflow vs stack overflow

So as a general rule to avoid a stack overflow, big objects should be allocated to the heap (correct me if I am wrong). But, since the heap and the stack expand towards each other, wouldn't this cause ...
0
votes
1answer
418 views

Priority queue for Kruskal's algorithm with running time O(E lg V)

I'm reviewing my notes on Kruskal's algorithm and have a question about getting the runtime down to O(E lg V). By using a PQ with edges and a boolean array of which vertices we have added to our tree ...
0
votes
1answer
352 views

What data structure is suitable for implementing dynamic huffman encoding and decoding on a piece of text?

Some pseudo code or resources will be appreciated.I was thinking if implementing it in form of a BST stored in an array. However,not all operations can be performed easily using this approach. I am ...
3
votes
3answers
284 views

Design pattern to dynamically create patterns found in a list of links

Can anyone help me think of a way to dynamically generate the patterns section of a tool I am creating? I'm not sure how to store and generate these "patterns" dynamically. What the program does is ...
3
votes
1answer
136 views

Algorithm/Strategy or Data structure to capture priorities sub priorities in an app

I am working on a CMS that is starting to evolve a bit. We started off with content that had the following priorities ( columnn in the db on the content table): HIGH MED LOW. Data was fetched by ...
1
vote
2answers
408 views

Shift bytes through a fixed sized buffer

I'm writing an InputStream that processes data streams containing "trailer" data. That is, the final n bytes of the stream is a piece of data that must be handled separately and shouldn't be returned ...
0
votes
1answer
355 views

Good Data Structure book to improve programming logic for children [closed]

My cousin (Age 10) who studies in Standard 5. He wants to learn programming and programming logic. Which book or resources should I refer? Is there any Good Data Structure book to improve ...
-1
votes
1answer
166 views

Storing User Form Data

I am trying to come up with a data schema/methodology for storing the data collected from various different user application forms. All forms contain the basic info like name, address, etc but then ...
-2
votes
2answers
270 views

About Artificial Intelligence [closed]

I am interested in starting a career in artificial intelligence. Can anyone suggest how I could prepare for this? What languages should I study that would be best for this career choice?
1
vote
1answer
141 views

Format text in a generic and reusable way

I would like to write some long text in some structure to allow a set of operations on that text. The question is which structure or format should I use, which suits best the use that I plan to do of ...
1
vote
3answers
1k views

The best way to store dictionary from file

I'm working on a translator in C++. Basically I want to parse the file with translations and store it in my program, so I can perform search through the words and simply access the corresponding word. ...
3
votes
2answers
200 views

Bayes filtering and data storage with expansive data sets

I am looking to write a Bayes filter that will act as an indicator of topic for a number of topics with a variable number of sources. Given a really big number of RSS feeds and here really big might ...
1
vote
1answer
297 views

Handling timeout in network application

How can I handle timeouts in a network application. I'm implementing a provisioning system on a Linux server, the code is huge so I'm going to put the algorithm, it works as like this Read ...
3
votes
4answers
1k views

Why many programming languages have only 2 data-structures: arrays and hashes?

Many programming languages have only those 2 structures, and even some languages that have more structures still only provide special syntax for those 2; usually, [] and {}. Why is this? Is there ...
3
votes
2answers
455 views

How should I structure a C# application that reads & writes binary data?

I have to read and write binary "chunks" of approximately 1Mb each. The data can come in the form of a stream or a in-memory byte[]. Normally I would use a Struct with a fixed layout, but there are ...
1
vote
3answers
352 views

What can Go chan do that a list cannot?

I want to know in which situation Go chan makes code much simpler than using list or queue or array that is usually available in all languages. As it was stated by Rob Pike in one of his speeches ...
4
votes
3answers
535 views

How Immutable Sets are Manipulated

When working with an immutable set or map, like the ones found in many functional programming languages, operations that would otherwise modify the container generate a new container instead. I know ...
9
votes
1answer
460 views

Optimal Data Structure for our own API

I'm in the early stages of writing an Emacs major mode for the Stack Exchange network; if you use Emacs regularly, this will benefit you in the end. In order to minimize the number of calls made to ...
2
votes
4answers
333 views

Plugin Architecture: How to Handle Databases and Logging?

I am in the process of designing an architecture for a plugin based application and I have some questions about how to handle database access and logging. The goal of my application is to allow a ...
12
votes
5answers
510 views

Storing in-text metadata in a discrete data structure

I am developing an application which will need to store inline, intext metadata. What I mean by that is the following: let's say we have a long text, and we want to store some metadata connected with ...
3
votes
3answers
284 views

Is it worth to try write foolproof data structures?

The problem We need to store data in a table-like way, but we have very strict space constraints (~1Mb per table of 10k+ rows). We store data like this: ID | reviews | factor | score | interval | ...
5
votes
6answers
3k views

Why is the usage of string keys generally considered to be a bad idea?

This has been bugging me for a while. Most of the time, when it comes to storing data in structures such as hashtables, programmers, books and articles insist that indexing elements in said structures ...
1
vote
1answer
173 views

Storage and naming of log data

I'm in the process of rewriting a small application from console mode to a GUI mode. The input is a file with log data from a measuring device: ... 00029;00044;00076;00044;00021 ...
3
votes
1answer
262 views

How do I transparently cache data in intermediate stages of processing?

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline. ckt = generate_ckt(ckt_properties); freq = generate_fpoints(fconfig); result = ...
2
votes
1answer
132 views

Applications with duplicate data - keep separate or combine into one set?

We currently have 3 similar, but slightly different applications that use the same data. We load the (same) data into each application that uses it. The applications are similar and use the same ...
6
votes
3answers
3k views

What is the best way to store a table in C++

I'm programming a decision tree in C++ using a slightly modified version of the C4.5 algorithm. Each node represents an attribute or a column of your data set and it has a children per possible value ...
0
votes
1answer
129 views

overworld vs on-screen 2d tile organization

I am developing a 2d game where the world is made up of tiles. I have an overworld and a "current world": the current world represent the current tiles on screen at any given time, currently this ...
1
vote
1answer
287 views

Database data structures for RESTful api

I am creating a RESTful API. I am struggling to decide on the best way to design my database tables around my resources. Initially, I though a table per resource would be a good way to go, but I'm ...
2
votes
2answers
2k views

Nested Enum type in C++ or C#?

I've come across a recurring issue in a few of my recent projects in which I find myself using enums to represent state, or type, or something else, and I need to check against a few conditions. Some ...
1
vote
1answer
276 views

Marked nodes in Fibonacci heaps

I don't understand why Fibonacci heaps have marked nodes (picture). A node is marked when its child is deleted. Quoting from Wikipedia: "[Keeping degree of each node low] is achieved by the rule that ...
2
votes
1answer
105 views

creating simple states for a stateless input

Given an input device (basically a keyboard) that reports keyup and keydown, how may I most efficiently store and retrieve information about which keys are currently depressed? My first thought was a ...
0
votes
2answers
219 views

Data structure for fast StartsWith filtering of a dictionary

I have a list of english words. I'm trying to come up with a data structure that allows me to do fast StartsWith filtering, like: var words = dictionary.StartsWith("foo"); I was thinking of ...
18
votes
2answers
2k views

What's the difference in content between Chris Okasaki's 1996 thesis and 1999 book, Purely Functional Data Structures?

I want to read Purely Functional Data Structures. I've easily found the thesis (which is freely available as a PDF), but see that there's a book available also. So I'd like to know what the ...