How to mount hadoop or hdfs file system in linux?

How to mount hadoop or hdfs file system in linux?

Purpose : To mount hdfs file in linux . It mainly use to dump the file and read only purpose .

Since hadoop-fuse-dfs is cloudera based solution , it is better to install both server and client installed from cloudera distribution itself

Steps :

1.Install hadoop on both servers and client :

2.Run the mount command in client

Detailed Steps :

A. Hadoop install I( in both client and server from cloudera distribution )

Steps :

1. Add cloudera distribution & Install hadoop-0.20-fuse ( this will install hadoop server)

wget http://archive.cloudera.com/redhat/6/x86_64/cdh/cdh3-repository-1.0-1.noarch.rpm

yum --nogpgcheck localinstall cdh3-repository-1.0-1.noarch.rpm

rpm --import http://archive.cloudera.com/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

yum install hadoop-0.20-fuse

2.set JAVA_HOME ( for both client and server)

hadoop-env.sh

Location:/usr/lib/hadoop-0.20/conf/

Export JAVA_HOME=

3. Configure hadoop in server and start

a. core-site.xml

Location :/usr/lib/hadoop-0.20/conf/

<property>

<name>hadoop.tmp.dir</name>

<value>/path/to/your/directory/hadoop-${user.name}</value>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://IP:9000</value>

</property>

b. Edit hdfs-site.xml

Location :/usr/lib/hadoop-0.20/conf/

<property>

  <name>dfs.replication</name>

  <value>1</value>

</property>

C. Edit  mapred-site.xml

Location :/usr/lib/hadoop-0.20/conf/

<property>

  <name>mapred.job.tracker</name>

  <value>master:9001</value>

</property>

d. Format Hadoop

hadoop namenode -format

e. start the hadoop

Location :/usr/lib/hadoop-0.20/bin

start-dfs.sh

start-mapred.sh

Use jps to verify the process

f. Use netstat to verify the hdfs port here :900

netstat -an | grep 9000

tcp 0 0 IP:9000 0.0.0.0:* LISTEN

tcp 0 0 IP:9000 10.184.37.158:45227 ESTABLISHED

4. In client

Set JAVA_HOME and create new folder for mount

a.run in debug mode to verify the connection

format :hadoop-fuse-dfs -d dfs://IP:9000 /home/hduser/mount/

$ hadoop-fuse-dfs -d dfs://IP:9000 /home/hduser/mount/

INFO fuse_options.c:116 Ignoring option -d

INFO fuse_options.c:165 Adding FUSE arg /home/hduser/mount/

FUSE library version: 2.8.3

nullpath_ok: 0

unique: 1, opcode: INIT (26), nodeid: 0, insize: 56

INIT: 7.20

hdfs getconf -confKey fs.default.name in server 

shows correct dfs location  . 

5 . Once mounting is fine ( it will show all the permission and users with out “?”)

then run the same command with out -d

Incorrect Mounting:

d?????????? ? ? ? ? ? mount

Correct Mounting

drwxr-xr-x. 2 hduser nobody 4096 Dec 31 1969 mount

Note : After mounting operation like copy or move works fine . But editing , appending the file is not possible .