Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...

learn more… | top users | synonyms

65
votes
6answers
5k views

Should temporary files be saved to /tmp or the current working directory?

I have a program that needs to generate temporary files. It is written for cluster machines. If I saved those files to a system-wide temporary directory (eg: /tmp), some users complained the program ...
62
votes
14answers
9k views

Are data type declarators like “int” and “char” stored in RAM when a C program executes?

When a C program is running, the data is stored on the heap or the stack. The values are stored in RAM addresses. But what about the type indicators (e.g., int or char)? Are they also stored? ...
41
votes
12answers
4k views

Is there an alternative to bits?

Is there an alternative to bits as the smallest unit of data? Something that won't be only 0 or 1, but actually hold many possible states in between? Wouldn't it be more natural to store floats like ...
32
votes
4answers
3k views

Should test data be checked into version control?

I'm writing some test code for a feature which processes PDF files. The basic idea behind the tests is that I point them towards some PDFs I've selected specially, they process them and I check that ...
27
votes
6answers
26k views

Why the static data members have to be defined outside the class separately in C++ (unlike Java)?

class A { static int foo () {} // ok static int x; // <--- needed to be defined separately in .cpp file }; I don't see a need of having A::x defined separately in a .cpp file (or same file ...
24
votes
2answers
14k views

Data Synchronization in mobile apps - multiple devices, multiple users

I'm looking into building my first mobile app. One of the core features of the application is that multiple devices/users will have access to the same data -- and all of them will have CRUD rights. I ...
20
votes
7answers
6k views

Why does Garbage Collection only sweep the heap?

Basically, I've learned so far that garbage collection erases forever any data structure that is not currently being pointed to. But this only checks the heap for such conditions. Why doesn't it also ...
20
votes
5answers
4k views

Data input validation - Where? How much? [closed]

Data input validation always was quite an internal struggle to me. On the verge of adding a real security framework and code to our legacy application rewrite project (which so far pretty much keeps ...
19
votes
1answer
371 views

How should HTML data formats be applied in everyday situations?

Given Google's shift towards a stronger focus on page markup data, how do the data formats used in Schema.org work alongside those for Microformats? How do these (and other specs) compliment each ...
16
votes
3answers
10k views

Is there a list of common usernames to reserve in a new system?

I need to reserve usernames in my new website. These generally fall into three categories 1) usernames no-one should have (eg: admin, user, service, help, root, etc) 2) names of super famous people ...
14
votes
8answers
1k views

Are there examples of non CRUD approaches?

I'm a programmer but also have worked as an archivist. As archivist it's a lot about keeping data. I often get into arguments with colleagues when it comes to operations on data. I don't like the U ...
12
votes
5answers
1k views

What's the proper term for a function inverse to a constructor - to unwrap a value from a data type?

Edit: I'm rephrasing the question a bit. Apparently I caused some confusion because I didn't realize that the term destructor is used in OOP for something quite different - it's a function invoked ...
11
votes
2answers
209 views

Should there be a “data hygiene” index for software - to indicate how clean the program is? not leaving temp files, etc

Should there be a "data hygiene" index for software - to indicate how clean the program is? not creating unused temp files, registry entries, environment variables, etc. For example, look in your ...
10
votes
5answers
32k views

Getting data from a webpage in a stable and efficient way

Recently I've learned that using a regex to parse the HTML of a website to get the data you need isn't the best course of action. So my question is simple: What then, is the best / most efficient and ...
9
votes
8answers
678 views

When does 'optimizing code' == 'structuring data'?

A recent article by ycombinator lists a comment with principles of a great programmer. #7. Good programmer: I optimize code. Better programmer: I structure data. Best programmer: What's the ...
8
votes
3answers
249 views

Is a single object to be preferred over multiple variables?

It was quite hard to put what I meant into a title, but it's easy to put into code. C++ Is this int offset_x = 10; int offset_y = 40; ... element.move(offset_x, offset_y); To be preferred over ...
8
votes
5answers
1k views

Sensitive Data Storage - Best Practices

I recently started working on a personal project where I was connecting to a database using Java. This got me thinking. I have to provide the login information for a database account on the DB server ...
8
votes
3answers
2k views

How do Data Scientists Use Python? [closed]

Do you know how Data Scientists use Python and if it requires data, where I can get sample data to practice with? I'm currently learning the language and plan on applying for Data Scientists jobs, so ...
8
votes
3answers
373 views

Data “unsort” / homogeneity algorithm

In an attempt to not reinvent a wheel, I'm asking if anyone has ideas on a data homogeneity algorithm. A brief example: My data has several elements maybe like Number Color Fruit Letter There ...
8
votes
5answers
3k views

How to decide between storage formats and what are example use cases for some of them?

We have different ways to store program data (save files in games, employee databases, program configuration etc.): Plain text (think .ini and .conf) XML Databases (MySQL, SQLite...) .zip and ...
7
votes
3answers
781 views

How to deal with constructors in large data classes [duplicate]

Several times now I have come across the situations where you have some kind of settings class that simply contains a mass of data. Often these classes are simply not valid without at least most of ...
7
votes
2answers
907 views

Why build data models in a dynamic language?

Background This question stems out of an architectural discussion we had at work. For the current project we are using Python and an object oriented database. There are input services that we ...
7
votes
1answer
450 views

Does current evidence support the adoption of Contextual over Canonical Data Models?

The "canonical" idea is pervasive in software; patterns like Canonical Model, Canonical Schema, Canonical Data Model and so on, seem to come up again and again in development. Like many developers, I'...
7
votes
2answers
1k views

Terminology: Really Delete vs set IsDeleted (“soft delete”) flag

I have a site where a regular user can delete records, where what happens is that an IsDeleted flag gets set. On administrative pages, a privileged user can delete records where an actual SQL DELETE ...
7
votes
3answers
297 views

Are there best practices for checking for errors in business logic for DB refreshes? [closed]

I'm working for a company with very large databases which all refresh overnight, ready for users to interrogate in the morning. These over night loads essentially truncate all the tables, and reload ...
7
votes
3answers
459 views

Microservices and isolated data storage models [duplicate]

I have been looking at Microservices for a while now. The concept is not new but it's communicated in a lightweight manner. So, I am very excited about this. However, there is a question that I am ...
6
votes
2answers
107 views

Seeking benefits of both SQL and binary files in a data storage scheme

I'm looking for some input on how to keep certain benefits of both a binary file and a SQL database in an interesting data storage problem. I have a legacy custom binary file format that is ...
6
votes
2answers
262 views

Data validation: separated class or not?

When I have a lot of data that needs to be validated, should I create a new class for the sole purpose of validation or should I stick with in-method validation? My particular example contemplates a ...
6
votes
3answers
217 views

Fuzzy algorithm for grouping data

I have data that indicates which topics each student has got right/wrong in an assessment.: | Student | Topic 1 | Topic 2 | Topic 3 | Topic 4 | Topic 5 | ... (perhaps 30 topics) | 1 | T | ...
6
votes
1answer
187 views

Is there a standard way of cleaning data files?

Having spent quite a lot of time working in software companies that deal in medium to large data sets, I've seen that a choke point in their processes is often prepping data files for loading into ...
6
votes
3answers
2k views

How to do data synchronisation between two applications?

Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database. Application A is used (...
5
votes
4answers
464 views

Can data classes contain methods for validation?

OK, say I have a data class for a user: public class User { private String firstName; private String lastName; private long personCode; private Date birthDate; private Gender gender; private String ...
5
votes
3answers
183 views

Why do people store shipping and handling in the order header and not a line item

I have worked on several ordering systems. Every one of them has tracked the shipping and handling as part of the order header rather then tracking it as a line item. To me, it would make more sense ...
5
votes
4answers
687 views

Randomly Accessing Data Packets in a Compressed File

In my line of work I deal with very large files, hundreds of Gigabytes in size. The nature of the data in these files is such that compression would greatly reduce their size. The problem is, the ...
5
votes
1answer
235 views

What are graph datastores better at doing than other datastores and why?

Where previously there was only relational databases, the data store world is now rife with options like Key-Value, Document, and Graph datastores. Unfortunately, every datastore likes to show how it ...
5
votes
4answers
781 views

Am I sending large amounts of data sensibly?

I am about to design a video conversion service, that is scalable on the conversion side. The architecture is as follows: Webpage for video upload When done, a message gets sent out to one of ...
5
votes
2answers
394 views

How to ensure data integrity (when an external system is involved)

Say you have this task in an application that consists of two parts: 1) Delivery of a file to an external system (not in your control) 2) A database transaction in our system, with information about ...
5
votes
1answer
164 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
5
votes
1answer
70 views

Bottleneck performance in ECS

I've been looking into building an entity-component-system. Basically, an entity is just an id wrapped around a struct, components are data belonging to that entity(and reference said id), and systems ...
5
votes
2answers
1k views

Deduplication of complex records / Similarity Detection

I'm working on a project that involves records with fairly large numbers of fields (~15-20) and I'm trying to figure out a good way to implement deduplication. Essentially the records are people along ...
5
votes
1answer
567 views

Data service API design

We're designing a data service that will store a large number of statistics about stocks, including historical data. The client will be a web application, which will need to pull various bits of data ...
5
votes
2answers
109 views

Resources on how to relate structured and semi- / un-structured information [closed]

I don't have a great background in information organisation / retrieval, but I know of a few ways of dealing with the problem. For structured information, it's possible to go OOish - everything "has-a"...
5
votes
0answers
2k views

How to parse multipart field/file data separately?

I want to parse a multipart form twice: Once to grab the incoming fields, and later to process the file upload. I am trying to maintain a proper separation of concerns within my Node app: ...
5
votes
6answers
454 views

Algorithm for determining transactions among weekly data series?

I'm trying to develop a small reporting tool (with sqlite backend). I can best describe this tool as a "transaction" ledger. What I'm trying to do is keep track of "transactions" from weekly data ...
4
votes
2answers
742 views

How can large data handling be a special requirement for job?

I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to ...
4
votes
5answers
664 views

Should I encrypt files saved by my program

I'm trying to decide on what format I should save data in, and hopefully this question will help me come to a decision. The program I'm writing uses 'databases' in which word=meaning (not a dictionary,...
4
votes
1answer
425 views

Should reflection be part of design?

I'm currently designing a somewhat large program that will involve the simulation of math/physics models and collection of data (I have not implemented any code yet). One of the main problems I'm ...
4
votes
2answers
339 views

Is this data model a list or tree?

Using python syntax with below environment diagram, teacher taught us that there are 11 trees(orange boundary) in this diagram, including leaf. It has been taught here, lists are represented as a ...
4
votes
3answers
706 views

Saving to Registry vs. CustomFile [closed]

I've been using the usual My.Settings... method when it came to saving settings for my program but I was just wondering, If I wanted to save program settings into the Registry and load those settings, ...
4
votes
2answers
600 views

Isolating test data in acceptance tests

I'm looking for guidance on how to keep my acceptance tests isolated. Right now the issue I'm having with being able to run the tests in parallel is the database records that are manipulated in the ...