How to mount hadoop
or hdfs file system in linux?
Purpose : To mount
hdfs file in linux . It mainly use to dump the file and read only
purpose .
Since
hadoop-fuse-dfs is cloudera based solution , it is better to install
both server and client installed from cloudera distribution itself
Steps :
1.Install hadoop on
both servers and client :
2.Run the mount
command in client
Detailed Steps :
A. Hadoop install
I( in both client and server from cloudera distribution )
Steps :
1. Add cloudera
distribution & Install hadoop-0.20-fuse ( this will install
hadoop server)
wget
http://archive.cloudera.com/redhat/6/x86_64/cdh/cdh3-repository-1.0-1.noarch.rpm
yum --nogpgcheck
localinstall cdh3-repository-1.0-1.noarch.rpm
yum install hadoop-0.20-fuse
2.set JAVA_HOME (
for both client and server)
hadoop-env.sh
Location:/usr/lib/hadoop-0.20/conf/
Export JAVA_HOME=
3.
Configure hadoop in server and start
a. core-site.xml
Location
:/usr/lib/hadoop-0.20/conf/
<property>
<name>hadoop.tmp.dir</name>
<value>/path/to/your/directory/hadoop-${user.name}</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://IP:9000</value>
</property>
b. Edit
hdfs-site.xml
Location
:/usr/lib/hadoop-0.20/conf/
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
C. Edit
mapred-site.xml
Location
:/usr/lib/hadoop-0.20/conf/
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
d.
Format Hadoop
hadoop namenode -format
e.
start the hadoop
Location :/usr/lib/hadoop-0.20/bin
start-dfs.sh
start-mapred.sh
Use jps to verify the process
f.
Use netstat to verify the hdfs port here :900
netstat -an | grep 9000
tcp 0 0 IP:9000 0.0.0.0:*
LISTEN
tcp 0 0 IP:9000 10.184.37.158:45227
ESTABLISHED
4. In client
Set JAVA_HOME and
create new folder for mount
a.run in debug
mode to verify the connection
format
:hadoop-fuse-dfs -d dfs://IP:9000 /home/hduser/mount/
$
hadoop-fuse-dfs -d dfs://IP:9000 /home/hduser/mount/
INFO
fuse_options.c:116 Ignoring option -d
INFO
fuse_options.c:165 Adding FUSE arg /home/hduser/mount/
FUSE library
version: 2.8.3
nullpath_ok: 0
unique: 1,
opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.20
shows correct dfs location .
5
. Once mounting is fine ( it will show all the permission and users
with out “?”)
then
run the same command with out -d
Incorrect
Mounting:
d?????????? ? ?
? ? ? mount
Correct
Mounting
drwxr-xr-x.
2 hduser nobody 4096 Dec 31 1969 mount
Note
: After mounting operation like copy or move works fine . But editing
, appending the file is not possible .
