Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...
38
votes
12answers
3k views
Is there an alternative to bits?
Is there an alternative to bits as the smallest unit of data? Something that won't be only 0 or 1, but actually hold many possible states inbetween? Wouldn't it be more natural to store floats like ...
20
votes
6answers
2k views
Why the static data members have to be defined outside the class separately in C++ (unlike Java)?
class A {
static int foo () {} // ok
static int x; // <--- needed to be defined separately in .cpp file
};
I don't see a need of having A::x defined separately in a .cpp file (or same file ...
19
votes
1answer
350 views
How should HTML data formats be applied in everyday situations?
Given Google's shift towards a stronger focus on page markup data, how do the data formats used in Schema.org work alongside those for Microformats? How do these (and other specs) compliment each ...
17
votes
7answers
1k views
Why does Garbage Collection only sweep the heap?
Basically, I've learned so far that garbage collection erases forever any data structure that is not currently being pointed to. But this only checks the heap for such conditions.
Why doesn't it also ...
17
votes
5answers
2k views
Data input validation - Where? How much?
Data input validation always was quite an internal struggle to me.
On the verge of adding a real security framework and code to our legacy application rewrite project (which so far pretty much keeps ...
11
votes
3answers
3k views
Is there a list of common usernames to reserve in a new system?
I need to reserve usernames in my new website.
These generally fall into three categories
1) usernames no-one should have (eg: admin, user, service, help, root, etc)
2) names of super famous people ...
10
votes
7answers
568 views
Are there examples of non CRUD approaches?
I'm a programmer but also have worked as an archivist. As archivist it's a lot about keeping data.
I often get into arguments with colleagues when it comes to operations on data. I don't like the U ...
10
votes
2answers
182 views
Should there be a “data hygiene” index for software - to indicate how clean the program is? not leaving temp files, etc
Should there be a "data hygiene" index for software - to indicate how clean the program is? not creating unused temp files, registry entries, environment variables, etc.
For example, look in your ...
9
votes
8answers
578 views
When does 'optimizing code' == 'structuring data'?
A recent article by ycombinator lists a comment with principles of a great programmer.
#7. Good programmer: I optimize code. Better programmer: I structure data. Best programmer: What's the ...
9
votes
5answers
1k views
How do Data Scientists Use Python?
Do you know how Data Scientists use Python and if it requires data, where I can get sample data to practice with? I'm currently learning the language and plan on applying for Data Scientists jobs, so ...
8
votes
11answers
3k views
Why do “data types” in computers exist, if it's really all just bits? [closed]
I am confused with the wide array of ways to implement existing data types in hardware as it all seems a bit illusory.
For example:
There's no such thing as a software stack or a hardware stack; a ...
8
votes
5answers
387 views
What's the proper term for a function inverse to a constructor - to unwrap a value from a data type?
Edit: I'm rephrasing the question a bit. Apparently I caused some confusion because I didn't realize that the term destructor is used in OOP for something quite different - it's a function invoked ...
8
votes
3answers
218 views
Is a single object to be preferred over multiple variables?
It was quite hard to put what I meant into a title, but it's easy to put into code.
C++
Is this
int offset_x = 10;
int offset_y = 40;
...
element.move(offset_x, offset_y);
To be preferred over ...
8
votes
3answers
287 views
Data “unsort” / homogeneity algorithm
In an attempt to not reinvent a wheel, I'm asking if anyone has ideas on a data homogeneity algorithm. A brief example:
My data has several elements maybe like
Number
Color
Fruit
Letter
There ...
8
votes
2answers
342 views
Why build data models in a dynamic language?
Background
This question stems out of an architectural discussion we had at work.
For the current project we are using Python and an object oriented database. There are input services that we ...
6
votes
4answers
382 views
Is a yobibit really a meaningful unit? [closed]
Wikipedia helpfully explains:
The yobibit is a multiple of the bit, a unit of digital information
storage, prefixed by the
standards-based multiplier yobi
(symbol Yi), a binary prefix ...
6
votes
5answers
487 views
Sensitive Data Storage - Best Practices
I recently started working on a personal project where I was connecting to a database using Java. This got me thinking. I have to provide the login information for a database account on the DB server ...
6
votes
5answers
2k views
Getting data from a webpage in a stable and efficient way
Recently I've learned that using a regex to parse the HTML of a website to get the data you need isn't the best course of action.
So my question is simple: What then, is the best / most efficient and ...
6
votes
4answers
224 views
Can data classes contain methods for validation?
OK, say I have a data class for a user:
public class User {
private String firstName;
private String lastName;
private long personCode;
private Date birthDate;
private Gender gender;
private String ...
6
votes
3answers
164 views
Are there best practices for checking for errors in business logic for DB refreshes?
I'm working for a company with very large databases which all refresh overnight, ready for users to interrogate in the morning. These over night loads essentially truncate all the tables, and reload ...
6
votes
1answer
230 views
Does current evidence support the adoption of Contextual over Canonical Data Models?
The "canonical" idea is pervasive in software; patterns like Canonical Model, Canonical Schema, Canonical Data Model and so on, seem to come up again and again in development.
Like many developers, ...
5
votes
5answers
465 views
How to decide between storage formats and what are example use cases for some of them?
We have different ways to store program data (save files in games, employee databases, program configuration etc.):
Plain text (think .ini and .conf)
XML
Databases (MySQL, SQLite...)
.zip and ...
5
votes
3answers
158 views
Why do people store shipping and handling in the order header and not a line item
I have worked on several ordering systems. Every one of them has tracked the shipping and handling as part of the order header rather then tracking it as a line item. To me, it would make more sense ...
5
votes
1answer
149 views
What are graph datastores better at doing than other datastores and why?
Where previously there was only relational databases, the data store world is now rife with options like Key-Value, Document, and Graph datastores. Unfortunately, every datastore likes to show how it ...
5
votes
4answers
349 views
Am I sending large amounts of data sensibly?
I am about to design a video conversion service, that is scalable on the conversion side.
The architecture is as follows:
Webpage for video upload
When done, a message gets sent out to one of ...
5
votes
1answer
171 views
Terminology: Really Delete vs set IsDeleted (“soft delete”) flag
I have a site where a regular user can delete records, where what happens is that an IsDeleted flag gets set. On administrative pages, a privileged user can delete records where an actual SQL DELETE ...
5
votes
2answers
74 views
Resources on how to relate structured and semi- / un-structured information
I don't have a great background in information organisation / retrieval, but I know of a few ways of dealing with the problem. For structured information, it's possible to go OOish - everything ...
4
votes
2answers
369 views
How can large data handling be a special requirement for job?
I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration.
I have worked on data migration to ...
4
votes
1answer
90 views
Isolating test data in acceptance tests
I'm looking for guidance on how to keep my acceptance tests isolated. Right now the issue I'm having with being able to run the tests in parallel is the database records that are manipulated in the ...
4
votes
2answers
165 views
A Question about using different data access layers for different database providers
I have a question about whether we should use different Data access layers for different database providers like SQL Server, MySQL Server, Oracle etc..
Is it not better to use a single data access ...
4
votes
2answers
157 views
How to ensure data integrity (when an external system is involved)
Say you have this task in an application that consists of two parts:
1) Delivery of a file to an external system (not in your control)
2) A database transaction in our system, with information about ...
4
votes
1answer
181 views
Data service API design
We're designing a data service that will store a large number of statistics about stocks, including historical data. The client will be a web application, which will need to pull various bits of data ...
4
votes
3answers
539 views
How to do data synchronisation between two applications?
Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database.
Application A is used ...
3
votes
5answers
193 views
make-like build tools for data?
Make is a standard tools for building software. But
make decides whether a target needs to be regenerated by comparing file modification times.
Are there any proven, preferably small tools that ...
3
votes
4answers
191 views
Reading Java book — I'm confused on this explanation of the difference between information and data?
From "Demystifying Java"..
For many of us, the terms information and data are synonymous. However, information and data are distinctly different in programming. Data is the smallest amount of ...
3
votes
3answers
207 views
Designing access to file-based “database”
It happened frequently that I have to provide access to a bunch of files organized in a directory tree according to some (sometimes loosely specified) rules. My standard pattern is to provide a ...
3
votes
3answers
734 views
Where can I find a large dataset of tweets?
I'm trying some machine learning with Twitter, and I was wondering if there were any pre-collected just massive tweet data that anyone knew about before I went and just used the API (which has a ...
3
votes
2answers
102 views
Design strategies for storing and validating serial numbers
We are writing software to track Foo Widgets. Each Foo Widget has a serial number. The serial number is an 32-character alphanumeric string. The string is separated into five sets. Each set is ...
3
votes
4answers
211 views
Randomly Accessing Data Packets in a Compressed File
In my line of work I deal with very large files, hundreds of Gigabytes in size. The nature of the data in these files is such that compression would greatly reduce their size. The problem is, the ...
3
votes
2answers
86 views
What's a good format for documenting hundreds of data feeds?
Several governments are starting to publish open data: datasets generated by the goverment, made freely available for citizens to use for value-added app development, analysis, and feedback. For ...
3
votes
1answer
193 views
Having a generic data type for a database table column, is it “good” practice?
I'm working on a PHP project where some object (class member) may contain different data type. For example :
class Property {
private $_id; // (PK)
private $_ref_id; // the object ...
3
votes
2answers
186 views
Getting and maintaining data from a large number of sources on the web
For the sake of an exercice I want to aggregate price listings with prices coming from a series of web sites in a structured form (XML, JSON...). If I have control on both sides how can I go about ...
3
votes
3answers
384 views
Where can I locate business data to use in my application?
This question talks about any and all free public raw data which appeared to have valuable pieces but nothing that really provides what I am looking for.
Instead of using a socially defined listing ...
3
votes
2answers
122 views
Design review for application facing memory issues
I apologise in advance for the length of this post, but I want to paint an accurate picture of the problems my app is facing and then pose some questions below;
I am trying to address some self ...
3
votes
1answer
189 views
Programming to ANSI standards (for engineering)
I am currently tasked to write a software to help engineers design standard compliant designs. If there is a bad design, software will report an error or warning.
Maybe it's just me, but anyone who ...
3
votes
4answers
916 views
Are the only types of data “sources” static and dynamic?
Thinking that there might be others, but not sure -- but before getting into that, let me explain what I mean by static and dynamic data sources.
Static (or datastore) - Meaning that the data's ...
3
votes
0answers
37 views
Looking for session/event analytics analysis tool
I have data that is basically made up of a set of sessions filled with events.
For example:
Session1 - [Event1] [Event2] [Event3]...[Event64]
Session2 - [Event1] [Event3] [Event4]...[Event64]
The ...
3
votes
1answer
139 views
How to process an endless XML data stream
There is an endless data stream of XML messages (and "heartbeats"), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel.
I'm still pondering. What is the best/most ...
2
votes
1answer
136 views
Dataset with coordinates of borders of countries
I am creating an application that visually displays world regions, e.g. to place markers within an administrative region.
Does a dataset exist with geometrical or geographical (long/lat) descriptions ...
2
votes
3answers
127 views
Having two sets of infrequently changing data or converting on the fly
I am currently working on a small project in which I work with some "static data", which is extracted from one source. The data might change from time to time (up to once a week).
I don't need all of ...
