Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...

learn more… | top users | synonyms

2
votes
1answer
104 views

What is the correct way to test performance of methods?

I made a few classes for processing and validating data in xlsx and csv. However, I am curious in testing how fast the methods I wrote in C# for the heavy lifting are. Here's what I currently am ...
1
vote
1answer
50 views

Handling Production Seed Data

We have a requirement for a new feature which requires some seed data to be present in the database (basically some default values) for the feature to function correctly. We have this in 2 different ...
-1
votes
0answers
38 views

Changing data in LVM format to CSV using Java?

So I need to take .lvm files -- which has a header and the data is separated by tabs -- and make them into .csv files with no header and data separated by commas. Am I able to do this using java? I ...
-1
votes
0answers
41 views

Gathering Specific Data from a Website - ways to do it? [closed]

I'm looking at a specific way to go through data on a website. What I want to do is Go to a page ( https://xrt.cfa.harvard.edu/missionops/timelines/) Open each file to get to the index for the date....
6
votes
2answers
107 views

Seeking benefits of both SQL and binary files in a data storage scheme

I'm looking for some input on how to keep certain benefits of both a binary file and a SQL database in an interesting data storage problem. I have a legacy custom binary file format that is ...
0
votes
0answers
23 views

What is needed server side in order to feed excel pivot users?

Our company has a lot of different business service "silos". They come with their client software, but they do not interop or offer cross domain searches. It is hard to compile the costs of a project ...
-3
votes
1answer
40 views

How to download data from a web page

I wonder if there is a way that I can get the data from a website automatically. Here's a sample link: http://www.trademap.org/CompaniesList.aspx?nvpm=1|704||||03|||4|1|1|2|3|1|1|1|1 You can ...
-1
votes
1answer
150 views

What is the faster way to save and browse objects using vectors +/or maps? (c++)

I'm doing a object Garbage class using c++: #include "Object.h" class ObjectGarbage { public: ObjectGarbage(); ~ObjectGarbage(); void Call_All_Objects() { //(...
3
votes
2answers
149 views

Correct way for storing data for runtime use in stand-alone applications?

I am working on a project that involves a "team builder" type application, if you will using C#. For the sake of simplicity, let us say it involves the user creating a "Team." There are three teams ...
-1
votes
1answer
107 views

What is the proper or recommended way to save String[]'s, Array's, ArrayList's and int[]'s to Android?

I've looked up this question many many times but keep seeing different answers, what is the BEST way to save array's? It seems ludicrously infantile that there is no standard or easy way to save ...
3
votes
5answers
427 views

What is the best way to save data of a C++ Program ? Binary vs JSON serialization

We are implementing a CAD program (in C++, Qt) where we have interdependant classes : The smallest brick is the Pattern, it is just a distribution of points. Then we have Layouts which contain ...
1
vote
2answers
55 views

Name for Keeping Multiple Copies of a Variable

There is a bad practice in software engineering where a single piece of data is held by value in multiple places. For example, in an FPS, the location of the player could be held by the AI class, the ...
4
votes
1answer
91 views

“Reactive programming is programming with asynchronous data streams.” is this true?

I was reading about Reactive programming, and came across this article The introduction to Reactive Programming you've been missing with the quote I put in the question title: Reactive programming ...
2
votes
1answer
67 views

Vector input for Artificial Neural Network?

My problem is fairly complex, so here is a similar, simplified example. Let's say you wanted to predict the Miles Per Gallon (fuel consumption) of a car, based on the following information: you have ...
1
vote
2answers
142 views

Designing validation classes

This is a follow-up of this question. I decided to implement what the accepted answer suggests, however, I realized I also needed individual public methods for each member the entity to be validated ...
6
votes
3answers
217 views

Fuzzy algorithm for grouping data

I have data that indicates which topics each student has got right/wrong in an assessment.: | Student | Topic 1 | Topic 2 | Topic 3 | Topic 4 | Topic 5 | ... (perhaps 30 topics) | 1 | T | ...
1
vote
0answers
31 views

Criteria for choosing which common data formats to support for tabular data in a simple ML library

What criteria does one consider when choosing which data formats a library for doing some machine learning task should support for reading/writing tabular (i.e. non-hierarchical) data? I found a ...
6
votes
2answers
262 views

Data validation: separated class or not?

When I have a lot of data that needs to be validated, should I create a new class for the sole purpose of validation or should I stick with in-method validation? My particular example contemplates a ...
-1
votes
1answer
57 views

How costly is DB data transfer as compared to data storage?

I've seen a number of answers (1, 2, 3) where they state that storage is cheap. So when designing a DB structure, it shouldn't be a main concern, or at least it doesn't rank very high in the list of ...
2
votes
4answers
206 views

What separates text files from binary files, considering they are stored in binary?

There are text files and there are image, video, and music files. Why are image, video, and music files considered binary files, and why are text files not?
0
votes
1answer
22 views

What relationship describes two different abstractions of the same dataset

I am wondering if there is a specific term for when multiple representations are data equivalent. Meaning that you can transfer the data from one representation to the other without any loss of data. ...
3
votes
3answers
164 views

How to test data based on SQL queries?

I'm trying to set up a test framework for (SQL) datasets that get updated daily. I want to write assertions that involve comparisons between two tables–for example, the sum of column visitors in the ...
1
vote
2answers
124 views

What design patterns exist for large dataset manipulation within MVC for a rich client data grid?

I want to feed an HTML5 data grid with large data sets that the client will want to manipulate in different ways, e.g. aggregation, grouping, pivoting. The data comes from different web services and ...
65
votes
6answers
5k views

Should temporary files be saved to /tmp or the current working directory?

I have a program that needs to generate temporary files. It is written for cluster machines. If I saved those files to a system-wide temporary directory (eg: /tmp), some users complained the program ...
1
vote
1answer
29 views

File structure for storing pixels that changed from one image to another

Have two images with the same dimensions. I want to create a file that contains information on what pixels are different between the two images. The file should also tell me the value of the new ...
0
votes
0answers
29 views

Storing mass user-files

I am thinking about the concepts of a web-application in which users can upload files to the server. I have multiple questions about the storage of these files. Imagine a service with 10,000+ users ...
3
votes
1answer
96 views

PHP OOP, structuring my objects properly

I have an architecture similar to this: User (Name, Email, ID...) ---- Studios (Name, Location, Description...) ---------- Videos (Title, Description, URL, Length, Views...) Simply put, a user can ...
1
vote
2answers
101 views

Best practice for creating defined versions in git

In my company we have a 'production pipeline', which is a bunch of code which we use to produce data, with the data later shipped to customers. The code is still being developed, bugs get fixed etc. ...
2
votes
2answers
71 views

How to perform summary analytics on production databases?

I want to use production data to generate "user activity" metrics. My thought is to set up a cron job that regularly reads production databases to summarize user activity and then stores the results ...
0
votes
0answers
37 views

Automated data processing in BPMN/BPEL?

In order to run complex simulations, I need to do preprocessing of data from various data sources. This is done by a bunch of Postgresql scripts. However, having only these is unsatisfactory, because ...
0
votes
1answer
103 views

Best practice for settings for PHP class for code generation [closed]

I would like to have a class creating any code from given options and output it in a chosen way. I have written class for generation of simple JavaScript code for such cases where is needed to ...
3
votes
1answer
60 views

Interactive review of data in modal before inserting in db

I'm looking for a conceptual approach to a problem. I'm building a web app that imports data from a source I don't control. The data quality is pretty high, but occasionally, the data does not make ...
7
votes
3answers
456 views

Microservices and isolated data storage models [duplicate]

I have been looking at Microservices for a while now. The concept is not new but it's communicated in a lightweight manner. So, I am very excited about this. However, there is a question that I am ...
1
vote
1answer
67 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
2
votes
2answers
353 views

Data Layer vs Business Layer

I think I'm in the minority on this but would be curious to see other perspectives. A lot of times I see people talk about the Data Layer like having functions like UpdateCustomer() or Summarize(). I ...
-1
votes
1answer
225 views

What is the difference between size and length? [closed]

It seems that the terms size and length are used interchangeably to describe how many bits, bytes or octets some data occupies, i.e. a length field in a data header is said to indicate the size of the ...
5
votes
1answer
164 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
0
votes
1answer
71 views

Better way to transform data [closed]

Context Imagine complicated service that provides info about bank products, fees and so on. Imagine complicated UI that deals with it. UI and Service data formats are changed 1-2 times/month because ...
2
votes
0answers
70 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
4
votes
2answers
51 views

AutoIncrement Integer Column,Guid or PGuid, which to choose?

Guid has been a Global Unique Identification. Why does "PGuid" (Global Unique Identification) appear? 1) Is Guid NOT PERMENT for unique? (I sometimes wonder whether Guid is REALLY PERMENT generator ...
7
votes
3answers
781 views

How to deal with constructors in large data classes [duplicate]

Several times now I have come across the situations where you have some kind of settings class that simply contains a mass of data. Often these classes are simply not valid without at least most of ...
3
votes
2answers
203 views

Design pattern for fetching data in chunks

I am creating Qt application that uses database with huge amount of data to draw some charts. Fetching data from the database is time consuming, so is blocking the application thread or worker thread ...
0
votes
0answers
24 views

Verify That Filters on a Dataset are Exclusive

Suppose I have a data table of, say, cars with a bunch of properties for each record. Example: |Vin|Year|Make|Model|Color|Mileage| I want to allow users to define (arbitrarily complex) filters which ...
2
votes
3answers
203 views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
1
vote
3answers
55 views

Find likely candidates for visited locations along gps track

I'm creating a visualization tool for workout data, specializing on bicycle rides. I'm looking for ways to generate more value from the data that was recorded. Since there usually is a GPS track ...
1
vote
0answers
82 views

Data Store Design for NxN Data Aggregation

I am trying to come up with a theoretical solution to an NxN problem for data aggregation and storage. As an example I have a huge amount of data that comes in via a stream. The stream sends the data ...
2
votes
3answers
94 views

Best practise for databinding when a non-data object is included with the data

I'm using .NET with Windows Forms, but I believe there may be similar concerns with other toolkits. Suppose we have a list of objects of a certain type. Using an example to make the following easier ...
3
votes
3answers
163 views

Storing great amounts of data

I was brainstorming and I found quite an interesting topic I couldn't come up with a solution for myself. How does storing excessively large amounts of data work? I mean physically, no matter how big ...
-1
votes
1answer
399 views

process large dataset c# [closed]

I have an application that process portfolios. I must aggregate 8 years of returns of aproximatelly 8K securities. (about 2016 doubles (8 years of data) for 2 dates (for comparison) 4032 for 8K ...
-2
votes
1answer
305 views

3D visualization of moment using accelerometer on Android device [closed]

Is it possible to reconstruct the movement of an Android device in 3D space using data from accelerometer? For example, I may draw a circle with my hand in the air, holding an Android device. Is ...