Is it possible to run Hadoop jobs (like the WordCount sample) in the local mode on Windows without Cygwin?

Question

I have Windows 7, Java 8, Maven and Eclipse. I\'ve created a Maven project and used almost exactly the same code as here.

It\'s just a simple \"word count\" sample.

Answer 1

Hadoop runs on Windows, it is possible, but you'll grow white hair if you try to pull it off on your own.

To start with, all filesystem operations in Windows Hadoop are routed either through the NativeIO, if available, or via winutils if NativeIO is not loaded. In your case it took the winutils path. You could make NativeIO available if you instruct Eclipse where to find it. See How to add native library to “java.library.path” with Eclipse launch (instead of overriding it), you need to add the location of that hadoop-common-project project target's bin, where you'll find hadoop.dll which hosts the NativeIO. But even after that, you'll still need wintils for container launch. The winutils.exe will be in that same location (the hadoop-common target/bin), but the code looks for it based on %HADOOP_HOME%, so you'll have to define that. And it will go uphill from there. I intentionally omitted the details how to configure all these because I don't think you should, or to be more precise, you should only if you understand how to do it.

It would be much much easier if you take an off-the-shelf Hadoop distribution for Windows, of which there are exactly one: the HDP from Hortonworks, download it, install it, configure it and then run against the 'cluster'.