Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Is anybody using Hadoop on Windows (Win32 or Win64) in production for serious work?

  • If you've tried it and rejected this combination can you give your (top) reasons?
  • If you managed to make it stable can you give an impression of how much work it took?

Background:

My company is 100% Windows. There's not much I can do about this other than use the tools I'm given.

According to the Hadoop documentation only UNIX-like platforms are recommended for production environments. Windows environments are currently considered to be experimental. I've tried some of the basic Hadoop Set-up on win32, and there seem to be a lot of gotchas, and I'm starting to suspect that this might be a foolish line of research.

I understand that Hadoop scripts depend to some extent on UNIX features such as Bash, Rsync, SSH which could be provided by Cygwin - however I'm not planning on becoming a pioneer. I'd like to run something which is somewhat standard. I want a decent chance of being able to get some community support if there's something I cannot solve all by myself.

I'm willing to invest some time to get this working on the company systems, but I'd like some reassurance from more pioneering types who have been there before. Alternatively I'd like some good advice from somebody who knows it's not worth bothering with!

share

migration rejected from stackoverflow.com Sep 25 '13 at 12:51

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as primarily opinion-based by gnat, GlenH7, MichaelT, Kilian Foth, Ozz Sep 25 '13 at 12:51

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

2  
I think you should better ask this on the public mailing list of hadoop. –  Thomas Jungblut Jul 5 '11 at 17:49
    
Thanks for the tip! I'm going to post the message in a number of relevant forums. –  Salim Fadhley Jul 5 '11 at 17:51

2 Answers 2

Looks like Microsoft has heard you. They made a big annoucement today that they will support Hadoop on Windows Server and Azure.

http://www.zdnet.com/blog/microsoft/microsoft-to-develop-hadoop-distributions-for-windows-server-and-azure/10958

share

EC2 or other cloud hosting is our answer. While corp. infrastructure can be pure windows, vms can be spun up to do work then shut down after the distributed job is completed. I wouldn't even try to get the traction on windows environment. Scalable makes it useful and a simple image can be used to easily spin up new nodes for you cluster in a way that would never work with windows.

share