The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
1answer
18 views

Best Azure Solution for Complex Search Index

I need to perform quick searches against a combination of tags while including date ranges: Example: Users who have requested notifications who did not respond to a notification sent at least 3 ...
-3
votes
1answer
126 views

Should I use NoSQL or HDFS for storage?

I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
2
votes
1answer
212 views

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average ...
1
vote
0answers
25 views

Offsetting high usage in a multi-tenant database environment?

I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database ...
0
votes
1answer
44 views

Does it make sense to store daily snapshots of data and analysis?

I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
2
votes
2answers
177 views

Relational vs Graph Database for (initially) moderately-sized network

We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beggining and have had no ...
0
votes
0answers
83 views

Best practices for dashboard of near real-time analytics

I’m currently building a dashboard to view some analytics about the data generated by my company's product. We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
0
votes
0answers
135 views

How to properly index MongoDB with ElasticSearch?

we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language). So we started to think about what kind of architecture could ...
0
votes
1answer
136 views

Feeding events to Elasticsearch, do I really need Logstash?

I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows: Visitor does something. Custom analytics server gathers data, makes an event out of it ...
2
votes
1answer
92 views

How to architect frequent updates in No-sql db (Cassandra) - architecture

I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content ...
1
vote
1answer
64 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
5
votes
1answer
160 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
6
votes
1answer
163 views

quantitatively comparing AST shapes

How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)? I guess that plagiarism detection on source code would use ...
2
votes
0answers
70 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
4
votes
4answers
161 views

Approaches for storing and analysing large amounts of time-based data

I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms. There are approx 250 data points (32-bit values), but only a subset ...
6
votes
4answers
1k views

Design of high performance file processing web application

I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering ...
0
votes
1answer
254 views

Is hadoop designed only for “simple” data processing jobs, where communications between the distributed nodes are sparse?

I am not a professional coder, but rather an engineer/mathematician that uses computer to solve numerical problems. So far most of my problems are math-related, such as solving large scale linear ...
2
votes
1answer
203 views

Random forest ML algorithm suitable for use on cluster based HPC?

I need help in identifying a better algorithm. I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve. The model contains over 12GB of data ...
0
votes
4answers
13k views

How to insert/update a large amount of data into mysql using php

I have an excel sheet which has a large amount of data. I am using php to insert the data into mysql server. I have two problems 1) I have to update a row if the id already exists, else insert the ...
4
votes
3answers
308 views

Big Data: Can it be pre-processed?

My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it. I would like to know: Whether or not large amounts of data ...
9
votes
3answers
6k views

Why Big Data Needs To Be Functional?

I started working on a new project lately related to Big Data for my internship. My managers recommended to start learning functional programming (They highly recommended Scala). I had a humbled ...
0
votes
2answers
157 views

finding patterns into logs

I am trying to put together a way to figure out if incidents occurred based on log content. Typically, on log (or db table) would contain a list of transactions composed as follow: {Timestamp} ...
1
vote
2answers
1k views

Data Aggregation of CSV files java

I have k csv files (5 csv files for example), each file has m fields which produce a key and n values. I need to produce a single csv file with aggregated data. I'm looking for the most efficient ...
14
votes
4answers
24k views

How to learn Cloud Computing and Big Data at home? [closed]

I want to learn Cloud Computing and Big Data at home. Is it possible to learn these technologies on home PC? Which technologies to learn in Cloud Computing? Which technologies to learn Big Data ...
1
vote
1answer
186 views

How to cross-reference many character encodings with ASCII OR UTFx?

I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the ...
0
votes
1answer
201 views

QFS (Quantcast File System): Scalable for medium “big data” problems or only for extremely large

I am doing some investigating and research into programming "big data", and am particularly interested in QuantCast (QFS) because it is written in my own favourite programming language, C++. Looking ...
1
vote
0answers
182 views

What's a good approach for updating millions of rows independently?

I have a MongoDB database with millions of users in collection. An aggregation function returns millions of results with the following format: Array ( [_id] => Array ( [user] => MongoId ...
2
votes
1answer
231 views

Experimenting with “big data” and map reduce

I was thinking about experimenting a bit with using mapreduce and such on a dataset we have to see if we can benefit from it in any way. What route should I go if I want to be able to instantly, ...
4
votes
5answers
2k views

Suggested Web Application Framework and Database for Enterprise, “Big-Data” App?

I have a web application that I have been developing for a small group within my company over the past few years, using Pipeline Pilot (plus jQuery and Python scripting) for web development and ...
23
votes
4answers
1k views

What is the definition of “Big Data”?

Is there one? All the definitions I can find describe the size, complexity / variety or velocity of the data. Wikipedia's definition is the only one I've found with an actual number Big data ...
5
votes
4answers
2k views

NoSQL and BIG DATA

I am doing an internship on Big Data technologies so I am new to this area. My question is about the use of NoSQL in the Big Data architecture. Do we need always to use a distributed storage (like ...
0
votes
2answers
735 views

Algorithms for data structures in distributed system

The hash table data structure can be easily spread across multiple machines with a simple algorithm to distribute the keys: machine_to_query = item_key % machine_count When you want to read and ...
7
votes
3answers
4k views

Python in Big Data?

Can python be efficiently implemented in big data field? To be precise I am building an web app that analyses really big data in medical health care field consisting of medical history and huge ...
3
votes
1answer
859 views

How does Google store search trends in backend?

Google trends shows what query has been searched how many times and some other properties of the said query. But how is this data stored in a database? Storing a new row for every search does not ...
1
vote
2answers
160 views

Using replacement to get possible outcomes to then search through HUGE amount of data

I have a database table holding 40 million records (table A). Each record has a string a user can search for. I also have a table with a list of character replacements (table B) i.e. i = Y, I = 1 ...
1
vote
1answer
1k views

Java BufferedReader vs Separate Producer consumer thread

I have a very big file delimited by some sequence of characters '*L*I*N*E'. The file will be of the order of 250G. And each line comes around 600bytes to 1000 bytes. I will be performing the following ...
1
vote
1answer
559 views

Developing an analytics's system processing large amounts of data - where to start

Imagine you're writing some sort of Web Analytics system - you're recording raw page hits along with some extra things like tagging cookies etc and then producing stats such as Which pages got most ...
3
votes
1answer
306 views

Tools for Operating on Large Amounts of Text Data

I need to handle and process ~5 million news articles, an amount small enough to fit on a commodity desktop but too large to process serially. Thus far, I have been using Python/pickle for ...
10
votes
3answers
2k views

Choose C++ or Java for applications requiring huge amounts of RAM? [closed]

I'm thinking of scientific applications that are mostly processor-bound and heavy on heap usage (at least several gigabytes). Any other time of the year I would happily go with C++, but in this case I ...