Hadoop release missing /conf directory

Question

I am trying to install a single node setup of Hadoop on Ubuntu. I started following the instructions on the Hadoop 2.3 docs.

But I seem to be missing something very simple.

First, it says to

To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.

Then,

Unpack the downloaded Hadoop distribution. In the distribution, edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation.

However, I can't seem to find the conf directory.

I downloaded a release of 2.3 at one of the mirrors. Then unpacked the tarball, an ls of the inside returns:

$ ls
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share

I was able to find the file they were referencing, just not in a conf directory:

$ find . -name hadoop-env.sh
./etc/hadoop/hadoop-env.sh

Am I missing something, or am I grabbing the wrong package? Or are the docs just outdated?

If so, anyone know where some more up-to date docs are?

Answer 1

I am trying to install a pseudo-distributed mode Hadoop, running into the same issue.

By following the book Hadoop The Definitive Guide (Third Edition), on page 618, it says:

In Hadoop 2.0 and later, MapReduce runs on YARN and there is an additional con-
figuration file called yarn-site.xml. All the configuration files should go in the
etc/hadoop subdirectory

Hope this confirms that etc/hadoop is the correct place.

Answer 2

I think the docs need to be updated. Although the directory structure has changed, file names for important files like hadoop-env.sh, core-ste.xml and hdfs-site.xml have not changed. You may find the following link useful for getting started.

http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html

Answer 3

In Hadoop1,

{$HADOOP_HOME}/conf/

In Hadoop2,

{$HADOOP_HOME}/etc/hadoop

Answer 4

Just adding a note on the blog post http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html. The blogpost is fantastic and very useful. That's how I got started. One aspect that I took a little time to figure is, that this blog seems to use a simplified way of providing configuration in the hadoop conf files such as "conf/core-site.xml", hdfs-site.xml etc... as follows

<!--fs.default.name is the name node URI -->
<configuration>
    fs.default.name
    hdfs://localhost:9000
</configuration>

As per official docs there is a more rigorous way - that would be useful when you have more than one properties is to add it as follows ( please note - the description is optional :-) )

<configuration>
    <property>
    <name> fs.default.name </name>
    <value>hdfs://localhost:9000 </value>
    <description>the name node URI </description>
    </property>
    <!--Add more configuration properties here -->
</configuration>

Answer 5

in Hadoop 2.7.3 the file is in hadoop-common/src/main/conf/

$ sudo find . -name hadoop-env.sh
./hadoop-2.7.3-src/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh