Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...

learn more… | top users | synonyms

-2
votes
0answers
67 views

Text file - editing in C++ [on hold]

I have some lines of text in a word doc. Each line has the format of an article, like: Author(s). Title. Journal, number of the Journal (Year), pages I want to put this data in Excel but in way such ...
1
vote
1answer
29 views

Consistency for disk write in a model object graph

You're writing an app where the model (as in MVC) is an object graph. Occasionally, you want to save this object graph to disk. What are some best practices to ensure that the write is consistent? (...
-2
votes
1answer
33 views

Need a solution for data filter problem

Problem statement: I have a input with multiple key value pairs and a list tasks stored in database with filter object. As an output I need a list of tasks matching the input values based on task's ...
15
votes
4answers
2k views

How should I store readonly data to deploy with my application?

I'm developing a desktop application, and this application requires some information to run, but it doesn't change any of this information (the data must be loaded on every execution of the app, but ...
-3
votes
1answer
84 views

What is 16 bit word's equivalent in decimal?

Question: For a 16 bit word with 6 bits for an opcode How many different instructions could I fit into the instruction set? What is the largest number that I could use as data? ...
-1
votes
2answers
97 views

better way to load data for millions of users

I am working on a Spring/JSF app in which i have 1 scheduler to load user's sleep activity from fitbit. Right now what i do is that whenever the scheduler kicks in, I load ALL users from mongoDB and ...
3
votes
1answer
136 views

What is the correct way to test performance of methods?

I made a few classes for processing and validating data in xlsx and csv. However, I am curious in testing how fast the methods I wrote in C# for the heavy lifting are. Here's what I currently am ...
1
vote
2answers
92 views

Handling Production Seed Data

We have a requirement for a new feature which requires some seed data to be present in the database (basically some default values) for the feature to function correctly. We have this in 2 different ...
6
votes
2answers
110 views

Seeking benefits of both SQL and binary files in a data storage scheme

I'm looking for some input on how to keep certain benefits of both a binary file and a SQL database in an interesting data storage problem. I have a legacy custom binary file format that is ...
0
votes
0answers
25 views

What is needed server side in order to feed excel pivot users?

Our company has a lot of different business service "silos". They come with their client software, but they do not interop or offer cross domain searches. It is hard to compile the costs of a project ...
-3
votes
1answer
41 views

How to download data from a web page

I wonder if there is a way that I can get the data from a website automatically. Here's a sample link: http://www.trademap.org/CompaniesList.aspx?nvpm=1|704||||03|||4|1|1|2|3|1|1|1|1 You can ...
-1
votes
1answer
150 views

What is the faster way to save and browse objects using vectors +/or maps? (c++)

I'm doing a object Garbage class using c++: #include "Object.h" class ObjectGarbage { public: ObjectGarbage(); ~ObjectGarbage(); void Call_All_Objects() { //(...
3
votes
2answers
164 views

Correct way for storing data for runtime use in stand-alone applications?

I am working on a project that involves a "team builder" type application, if you will using C#. For the sake of simplicity, let us say it involves the user creating a "Team." There are three teams ...
-1
votes
1answer
116 views

What is the proper or recommended way to save String[]'s, Array's, ArrayList's and int[]'s to Android?

I've looked up this question many many times but keep seeing different answers, what is the BEST way to save array's? It seems ludicrously infantile that there is no standard or easy way to save ...
3
votes
5answers
455 views

What is the best way to save data of a C++ Program ? Binary vs JSON serialization

We are implementing a CAD program (in C++, Qt) where we have interdependant classes : The smallest brick is the Pattern, it is just a distribution of points. Then we have Layouts which contain ...
1
vote
2answers
56 views

Name for Keeping Multiple Copies of a Variable

There is a bad practice in software engineering where a single piece of data is held by value in multiple places. For example, in an FPS, the location of the player could be held by the AI class, the ...
4
votes
1answer
99 views

“Reactive programming is programming with asynchronous data streams.” is this true?

I was reading about Reactive programming, and came across this article The introduction to Reactive Programming you've been missing with the quote I put in the question title: Reactive programming ...
2
votes
1answer
73 views

Vector input for Artificial Neural Network?

My problem is fairly complex, so here is a similar, simplified example. Let's say you wanted to predict the Miles Per Gallon (fuel consumption) of a car, based on the following information: you have ...
1
vote
2answers
167 views

Designing validation classes

This is a follow-up of this question. I decided to implement what the accepted answer suggests, however, I realized I also needed individual public methods for each member the entity to be validated ...
6
votes
3answers
218 views

Fuzzy algorithm for grouping data

I have data that indicates which topics each student has got right/wrong in an assessment.: | Student | Topic 1 | Topic 2 | Topic 3 | Topic 4 | Topic 5 | ... (perhaps 30 topics) | 1 | T | ...
1
vote
0answers
31 views

Criteria for choosing which common data formats to support for tabular data in a simple ML library

What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a ...
6
votes
2answers
318 views

Data validation: separated class or not?

When I have a lot of data that needs to be validated, should I create a new class for the sole purpose of validation or should I stick with in-method validation? My particular example contemplates a ...
-1
votes
1answer
57 views

How costly is DB data transfer as compared to data storage?

I've seen a number of answers (1, 2, 3) where they state that storage is cheap. So when designing a DB structure, it shouldn't be a main concern, or at least it doesn't rank very high in the list of ...
2
votes
4answers
214 views

What separates text files from binary files, considering they are stored in binary?

There are text files and there are image, video, and music files. Why are image, video, and music files considered binary files, and why are text files not?
0
votes
1answer
22 views

What relationship describes two different abstractions of the same dataset

I am wondering if there is a specific term for when multiple representations are data equivalent. Meaning that you can transfer the data from one representation to the other without any loss of data. ...
3
votes
3answers
222 views

How to test data based on SQL queries?

I'm trying to set up a test framework for (SQL) datasets that get updated daily. I want to write assertions that involve comparisons between two tables–for example, the sum of column visitors in the ...
1
vote
2answers
158 views

What design patterns exist for large dataset manipulation within MVC for a rich client data grid?

I want to feed an HTML5 data grid with large data sets that the client will want to manipulate in different ways, e.g. aggregation, grouping, pivoting. The data comes from different web services and ...
65
votes
6answers
5k views

Should temporary files be saved to /tmp or the current working directory?

I have a program that needs to generate temporary files. It is written for cluster machines. If I saved those files to a system-wide temporary directory (eg: /tmp), some users complained the program ...
1
vote
1answer
30 views

File structure for storing pixels that changed from one image to another

Have two images with the same dimensions. I want to create a file that contains information on what pixels are different between the two images. The file should also tell me the value of the new ...
0
votes
0answers
30 views

Storing mass user-files

I am thinking about the concepts of a web-application in which users can upload files to the server. I have multiple questions about the storage of these files. Imagine a service with 10,000+ users ...
3
votes
1answer
99 views

PHP OOP, structuring my objects properly

I have an architecture similar to this: User (Name, Email, ID...) ---- Studios (Name, Location, Description...) ---------- Videos (Title, Description, URL, Length, Views...) Simply put, a user can ...
1
vote
2answers
102 views

Best practice for creating defined versions in git

In my company we have a 'production pipeline', which is a bunch of code which we use to produce data, with the data later shipped to customers. The code is still being developed, bugs get fixed etc. ...
2
votes
2answers
76 views

How to perform summary analytics on production databases?

I want to use production data to generate "user activity" metrics. My thought is to set up a cron job that regularly reads production databases to summarize user activity and then stores the results ...
0
votes
0answers
37 views

Automated data processing in BPMN/BPEL?

In order to run complex simulations, I need to do preprocessing of data from various data sources. This is done by a bunch of Postgresql scripts. However, having only these is unsatisfactory, because ...
0
votes
1answer
105 views

Best practice for settings for PHP class for code generation [closed]

I would like to have a class creating any code from given options and output it in a chosen way. I have written class for generation of simple JavaScript code for such cases where is needed to ...
3
votes
1answer
60 views

Interactive review of data in modal before inserting in db

I'm looking for a conceptual approach to a problem. I'm building a web app that imports data from a source I don't control. The data quality is pretty high, but occasionally, the data does not make ...
7
votes
3answers
536 views

Microservices and isolated data storage models [duplicate]

I have been looking at Microservices for a while now. The concept is not new but it's communicated in a lightweight manner. So, I am very excited about this. However, there is a question that I am ...
1
vote
1answer
68 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
2
votes
2answers
375 views

Data Layer vs Business Layer

I think I'm in the minority on this but would be curious to see other perspectives. A lot of times I see people talk about the Data Layer like having functions like UpdateCustomer() or Summarize(). I ...
-1
votes
1answer
254 views

What is the difference between size and length? [closed]

It seems that the terms size and length are used interchangeably to describe how many bits, bytes or octets some data occupies, i.e. a length field in a data header is said to indicate the size of the ...
5
votes
1answer
164 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
0
votes
1answer
75 views

Better way to transform data [closed]

Context Imagine complicated service that provides info about bank products, fees and so on. Imagine complicated UI that deals with it. UI and Service data formats are changed 1-2 times/month because ...
2
votes
0answers
71 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
4
votes
2answers
52 views

AutoIncrement Integer Column,Guid or PGuid, which to choose?

Guid has been a Global Unique Identification. Why does "PGuid" (Global Unique Identification) appear? 1) Is Guid NOT PERMENT for unique? (I sometimes wonder whether Guid is REALLY PERMENT generator ...
7
votes
3answers
796 views

How to deal with constructors in large data classes [duplicate]

Several times now I have come across the situations where you have some kind of settings class that simply contains a mass of data. Often these classes are simply not valid without at least most of ...
3
votes
2answers
211 views

Design pattern for fetching data in chunks

I am creating Qt application that uses database with huge amount of data to draw some charts. Fetching data from the database is time consuming, so is blocking the application thread or worker thread ...
0
votes
0answers
25 views

Verify That Filters on a Dataset are Exclusive

Suppose I have a data table of, say, cars with a bunch of properties for each record. Example: |Vin|Year|Make|Model|Color|Mileage| I want to allow users to define (arbitrarily complex) filters which ...
2
votes
3answers
213 views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
1
vote
3answers
56 views

Find likely candidates for visited locations along gps track

I'm creating a visualization tool for workout data, specializing on bicycle rides. I'm looking for ways to generate more value from the data that was recorded. Since there usually is a GPS track ...
1
vote
0answers
84 views

Data Store Design for NxN Data Aggregation

I am trying to come up with a theoretical solution to an NxN problem for data aggregation and storage. As an example I have a huge amount of data that comes in via a stream. The stream sends the data ...