The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

learn more… | top users | synonyms

-3
votes
0answers
93 views

Switch career from Java to big data - confused :/ [closed]

I need some information to take a decision which matters a lot to me. I know this is a bit lengthy and I request you to read through it. Currently I am working as an application developer with the ...
3
votes
1answer
206 views

Hadoop and Object Reuse, Why?

In Hadoop, objects passed to reducers are reused. This is extremely surprising and hard to track down if you're not expecting it. Furthermore, the original tracker for this "feature" doesn't offer any ...
1
vote
0answers
124 views

Which are the cons of a purely stream-based architecture against a Lambda architecture? [closed]

Disclaimer: I'm not a real-time architectures expert, I'd like only to throw a couple of personal considerations and evaluate what others would suggest or point out. Let's imagine we'd like to design ...
3
votes
3answers
924 views

Why do HDFS clusters have only a single NameNode?

I'm trying to understand better how Hadoop works, and I'm reading The NameNode is a Single Point of Failure for the HDFS Cluster. HDFS is not currently a High Availability system. When the ...
4
votes
6answers
9k views

What is the value of the Cloudera Hadoop Certification for people new to the IT industry? [closed]

I am a software developer with 8 months of experience in the IT industry, currently working on the development of tools for BIG DATA analytics. I have learned Hadoop basics on my own and I am pretty ...
0
votes
0answers
66 views

Searching in a large dataset

I am working on a model of social interactions in mice. I have mice and boxes and a simulation that outputs which mouse stays in which box during which time period. The problem is how to obtain, in ...
2
votes
1answer
420 views

How best to merge/sort/page through tons of JSON arrays?

Here's the scenario: Say you have millions of JSON documents stored as text files. Each JSON document is an array of "activity" objects, each of which contain a "created_datetime" attribute. What is ...
2
votes
1answer
671 views

How best to implement a Dashboard from data in HDFS/Hadoop [closed]

We have a bunch of data (several TB) in Hadoop HDFS and it's growing. We want to create a dashboard that reports on the contents in there e.g counts of different types of objects, trends over time ...
2
votes
3answers
538 views

Scalable distributed file system for blobs like images and other documents [closed]

Cassandra & HBase both do not efficiently support storage of blobs like images. Storing directly on HDFS stresses the Namenode because of huge number of files. Facebook uses Haystack for images ...
3
votes
2answers
374 views

Can Hadoop be made production-stable on a Windows platform? [closed]

Is anybody using Hadoop on Windows (Win32 or Win64) in production for serious work? If you've tried it and rejected this combination can you give your (top) reasons? If you managed to make it ...
4
votes
2answers
897 views

how to convince other we should move to hadoop?

Everything I've read about Hadoop seems like exactly the technology we need to make our enterprise more scalable. We have terabytes of raw data that is in non-relational form (text files of some ...
12
votes
1answer
11k views

What is Hadoop and what are some example applications of it? [closed]

The title basically says it all. I keep hearing about Hadoop, and from the wikipedia entry, it certainly sounds interesting, but I'm curious if there are some good examples where it makes sense to ...
3
votes
2answers
454 views

Text search - big data problem

I have a problem I was hoping I could get some advice on! I have a LOT of text as input (about 20GB worth, not MASSIVE but big enough). This is just free text, unstructured. I have a 'category ...
3
votes
3answers
14k views

Is cloudera hadoop certification worth the investment [duplicate]

I am considering investing time to learn Hadoop and it's related technologies. The problem is that my current day job will not be using Hadoop any time soon and even if I learn from books, blogs ...
5
votes
2answers
1k views

Optimal way to store 18 billion key, value pairs [closed]

I have around 200 million new objects coming in, and a 90 day retention policy, so that leaves me with 18 billion records to be stored in the form of key-value pairs. Key and value both will be a ...
1
vote
1answer
188 views

Design application to send messages by marking circle on the map where you want to send message? [closed]

Design application to send messages by marking circle on the map where you want to send message ? This is a question that was asked of me by an interviewer. Given a map of the world, Select ...
1
vote
1answer
279 views

Is it smart to design a command and control server, that will monitor system resources and spin up/spin down servers at times of peak?

I am building an application that will be modular, in a way that it will be a set of separate systems communicating with each other. It uses Hadoop on all systems, and HBase on 3 of the 4. Scaling ...
4
votes
2answers
495 views

Asynchronous Java

I'm wondering if I wanted to implement a web service based on java that does web analytics, what sort of architecture should I use. The actualy processing of the Big Data would be done by Hadoop. ...
7
votes
2answers
924 views

Can map-reduce say “Hello World”?

Gathering that map-reduce is being used to process huge amounts of data, I set out to understand it. My queries were: What class of problems does it aim to solve? How does it help breaking down of ...
1
vote
1answer
107 views

How do you control nodes in a server farm?

I've been reading about hadoop and multi-node setups, and it says in the documentation that you must have a JVM and hadoop software already running on those nodes. My question is, do people install ...