Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...

learn more… | top users | synonyms

0
votes
2answers
11 views

Name for Keeping Multiple Copies of a Variable

There is a bad practice in software engineering where a single piece of data is held by value in multiple places. For example, in an FPS, the location of the player could be held by the AI class, the ...
3
votes
1answer
70 views

“Reactive programming is programming with asynchronous data streams.” is this true?

I was reading about Reactive programming, and came across this article The introduction to Reactive Programming you've been missing with the quote I put in the question title: Reactive programming ...
2
votes
1answer
55 views

Vector input for Artificial Neural Network?

My problem is fairly complex, so here is a similar, simplified example. Let's say you wanted to predict the Miles Per Gallon (fuel consumption) of a car, based on the following information: you have ...
1
vote
1answer
66 views

Designing validation classes

This is a follow-up of this question. I decided to implement what the accepted answer suggests, however, I realized I also needed individual public methods for each member the entity to be validated ...
6
votes
1answer
86 views

Fuzzy algorithm for grouping data

I have data that indicates which topics each student has got right/wrong in an assessment.: | Student | Topic 1 | Topic 2 | Topic 3 | Topic 4 | Topic 5 | ... (perhaps 30 topics) | 1 | T | ...
0
votes
0answers
26 views

Criteria for choosing which common data formats to support for tabular data in a simple ML library

What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a ...
6
votes
2answers
208 views

Data validation: separated class or not?

When I have a lot of data that needs to be validated, should I create a new class for the sole purpose of validation or should I stick with in-method validation? My particular example contemplates a ...
-1
votes
1answer
55 views

How costly is DB data transfer as compared to data storage?

I've seen a number of answers (1, 2, 3) where they state that storage is cheap. So when designing a DB structure, it shouldn't be a main concern, or at least it doesn't rank very high in the list of ...
2
votes
4answers
197 views

What separates text files from binary files, considering they are stored in binary?

There are text files and there are image, video, and music files. Why are image, video, and music files considered binary files, and why are text files not?
0
votes
1answer
17 views

What relationship describes two different abstractions of the same dataset

I am wondering if there is a specific term for when multiple representations are data equivalent. Meaning that you can transfer the data from one representation to the other without any loss of data. ...
3
votes
3answers
115 views

How to test data based on SQL queries?

I'm trying to set up a test framework for (SQL) datasets that get updated daily. I want to write assertions that involve comparisons between two tables–for example, the sum of column visitors in the ...
1
vote
2answers
65 views

What design patterns exist for large dataset manipulation within MVC for a rich client data grid?

I want to feed an HTML5 data grid with large data sets that the client will want to manipulate in different ways, e.g. aggregation, grouping, pivoting. The data comes from different web services and ...
64
votes
6answers
5k views

Should temporary files be saved to /tmp or the current working directory?

I have a program that needs to generate temporary files. It is written for cluster machines. If I saved those files to a system-wide temporary directory (eg: /tmp), some users complained the program ...
1
vote
1answer
29 views

File structure for storing pixels that changed from one image to another

Have two images with the same dimensions. I want to create a file that contains information on what pixels are different between the two images. The file should also tell me the value of the new ...
0
votes
0answers
27 views

Storing mass user-files

I am thinking about the concepts of a web-application in which users can upload files to the server. I have multiple questions about the storage of these files. Imagine a service with 10,000+ users ...
3
votes
1answer
91 views

PHP OOP, structuring my objects properly

I have an architecture similar to this: User (Name, Email, ID...) ---- Studios (Name, Location, Description...) ---------- Videos (Title, Description, URL, Length, Views...) Simply put, a user can ...
0
votes
1answer
105 views

Database vs. Pickle (Python) [closed]

I'm writing a small application in Python that stores data via pickle. Initially, I was trying to decide between pickle and a database solution, and went with pickle to try something new. What I want ...
1
vote
2answers
100 views

Best practice for creating defined versions in git

In my company we have a 'production pipeline', which is a bunch of code which we use to produce data, with the data later shipped to customers. The code is still being developed, bugs get fixed etc. ...
2
votes
2answers
54 views

How to perform summary analytics on production databases?

I want to use production data to generate "user activity" metrics. My thought is to set up a cron job that regularly reads production databases to summarize user activity and then stores the results ...
0
votes
0answers
34 views

Automated data processing in BPMN/BPEL?

In order to run complex simulations, I need to do preprocessing of data from various data sources. This is done by a bunch of Postgresql scripts. However, having only these is unsatisfactory, because ...
0
votes
1answer
95 views

Best practice for settings for PHP class for code generation [closed]

I would like to have a class creating any code from given options and output it in a chosen way. I have written class for generation of simple JavaScript code for such cases where is needed to ...
3
votes
1answer
60 views

Interactive review of data in modal before inserting in db

I'm looking for a conceptual approach to a problem. I'm building a web app that imports data from a source I don't control. The data quality is pretty high, but occasionally, the data does not make ...
7
votes
3answers
334 views

Microservices and isolated data storage models [duplicate]

I have been looking at Microservices for a while now. The concept is not new but it's communicated in a lightweight manner. So, I am very excited about this. However, there is a question that I am ...
1
vote
1answer
63 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
2
votes
2answers
296 views

Data Layer vs Business Layer

I think I'm in the minority on this but would be curious to see other perspectives. A lot of times I see people talk about the Data Layer like having functions like UpdateCustomer() or Summarize(). I ...
-1
votes
1answer
177 views

What is the difference between size and length? [closed]

It seems that the terms size and length are used interchangeably to describe how many bits, bytes or octets some data occupies, i.e. a length field in a data header is said to indicate the size of the ...
6
votes
1answer
157 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
0
votes
1answer
64 views

Better way to transform data [closed]

Context Imagine complicated service that provides info about bank products, fees and so on. Imagine complicated UI that deals with it. UI and Service data formats are changed 1-2 times/month because ...
2
votes
0answers
70 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
4
votes
2answers
51 views

AutoIncrement Integer Column,Guid or PGuid, which to choose?

Guid has been a Global Unique Identification. Why does "PGuid" (Global Unique Identification) appear? 1) Is Guid NOT PERMENT for unique? (I sometimes wonder whether Guid is REALLY PERMENT generator ...
7
votes
3answers
760 views

How to deal with constructors in large data classes [duplicate]

Several times now I have come across the situations where you have some kind of settings class that simply contains a mass of data. Often these classes are simply not valid without at least most of ...
2
votes
2answers
179 views

Design pattern for fetching data in chunks

I am creating Qt application that uses database with huge amount of data to draw some charts. Fetching data from the database is time consuming, so is blocking the application thread or worker thread ...
0
votes
0answers
24 views

Verify That Filters on a Dataset are Exclusive

Suppose I have a data table of, say, cars with a bunch of properties for each record. Example: |Vin|Year|Make|Model|Color|Mileage| I want to allow users to define (arbitrarily complex) filters which ...
2
votes
3answers
189 views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
1
vote
3answers
51 views

Find likely candidates for visited locations along gps track

I'm creating a visualization tool for workout data, specializing on bicycle rides. I'm looking for ways to generate more value from the data that was recorded. Since there usually is a GPS track ...
1
vote
0answers
76 views

Data Store Design for NxN Data Aggregation

I am trying to come up with a theoretical solution to an NxN problem for data aggregation and storage. As an example I have a huge amount of data that comes in via a stream. The stream sends the data ...
2
votes
3answers
91 views

Best practise for databinding when a non-data object is included with the data

I'm using .NET with Windows Forms, but I believe there may be similar concerns with other toolkits. Suppose we have a list of objects of a certain type. Using an example to make the following easier ...
3
votes
3answers
162 views

Storing great amounts of data

I was brainstorming and I found quite an interesting topic I couldn't come up with a solution for myself. How does storing excessively large amounts of data work? I mean physically, no matter how big ...
-1
votes
1answer
306 views

process large dataset c# [closed]

I have an application that process portfolios. I must aggregate 8 years of returns of aproximatelly 8K securities. (about 2016 doubles (8 years of data) for 2 dates (for comparison) 4032 for 8K ...
-2
votes
1answer
242 views

3D visualization of moment using accelerometer on Android device [closed]

Is it possible to reconstruct the movement of an Android device in 3D space using data from accelerometer? For example, I may draw a circle with my hand in the air, holding an Android device. Is ...
2
votes
2answers
151 views

How to handle multiple models that relate to a single model

First of, sorry for the vague title. In our website framework, we have modal classes to represent our database tables, that hold a rows data. I'm currently working on a stat style page, and I'm ...
1
vote
1answer
63 views

Send more data, or do client processing?

I have a conceptual problem that I want feedback on. First and foremost, the layout of my system. It contains a server, with many clients connected to it. The clients get fed data from the server at ...
5
votes
6answers
427 views

Algorithm for determining transactions among weekly data series?

I'm trying to develop a small reporting tool (with sqlite backend). I can best describe this tool as a "transaction" ledger. What I'm trying to do is keep track of "transactions" from weekly data ...
1
vote
0answers
86 views

Saving and retrieving multiple cached json/txt data files

I am working on some numerical programming and need to generate the results for a model given a variety of input parameters. Since the model takes a while to run, I was planning to save the data to ...
2
votes
2answers
332 views

How do you sync changes to the data in your database? [closed]

I am new to a team developing a very large web application. Prior to my start, the team had 0 automation in place and just did everything by hand. This was possible because they were only a team of 2. ...
1
vote
0answers
89 views

Using a distributed system as an alternative to disk storage

So, one of the differences between "main memory" and disk storage is durability -- the data on hard disks will stay accessible and unaffected even if the host machine crashes or loses power. One can ...
4
votes
1answer
112 views

What method should I use to populate a tree as I'm consuming data that is contained within the tree?

I have a tree that consists of 5 types of nodes - V, W, X, Y, and Z. V represents the root node, and a given tree has exactly one element of type V. A V contains one or more Ws, a W contains one or ...
60
votes
14answers
9k views

Are data type declarators like “int” and “char” stored in RAM when a C program executes?

When a C program is running, the data is stored on the heap or the stack. The values are stored in RAM addresses. But what about the type indicators (e.g., int or char)? Are they also stored? ...
0
votes
3answers
2k views

Is it bad to store data in JSON form or should I use mySQL? [closed]

I've been building an application and the whole thing is built around a simple data storage system. Create an array. Convert it into JSON. Add <?php die() at the beginning to prevent access. ...
1
vote
2answers
199 views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...