java.lang.ClassNotFoundException and No FileSystem for scheme: hdfs

java.lang.ClassNotFoundException and No FileSystem for scheme: hdfs  exception while connecting to hadoop from application server



Stand alone code to connect to hadoop works fine But when the same code put it in application server ( weblogic ) , it fails . Even though all the jars bundled in ear file


Error 1 :

java.io.IOException: No FileSystem for scheme: hdfs                                                                        
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1600)                                          
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:69)                                                  
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1637)                                        
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1619)                                                
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:255)                                                        
        at connector.HadoopService.hadoopHandler(HadoopService.java:62)                                                    
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                    
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)                                  
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingM


Fix :
Edit the code  like this

                      Configuration conf = new Configuration();
               

              conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
             conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());



        FileSystem hdfs = FileSystem.get(new URI("hdfs://IP:9000"),conf);




Error 2 :


java.lang.ClassNotFoundException: Class  org.apache.hadoop.hdfs.DistributedFileSystem
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2290)
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2303)
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)




Fix : add all the jars required specifically in <domain>/lib directory

For hadoop 2.7.2  : required jars  ( all are located at common/lib folder )

1. commons-io-2.4.jar       
2.   guava-11.0.2.jar    
3.    hadoop-common-2.7.2.jar 
4.  htrace-core-3.1.0-incubating.jar 
5.  protobuf-java-2.5.0.jar 
6.  slf4j-api-1.7.10.jar
7. commons-logging-1.1.3.jar 
8.  hadoop-auth-2.7.2.jar 
9.  hadoop-hdfs-2.7.2.jar  
10.   log4j-1.2.17.jar   

For coludera distribution 

1. log4j-1.2.17.jar

Location : http://redrockdigimark.com/apachemirror/logging/log4j/1.2.17/log4j-1.2.17.zip

2. commons-logging-1.0.4.jar

3. guava-r09-jarjar.jar

Location : http://www.java2s.com/Code/JarDownload/guava/guava-r09-jarjar.jar.zip

4. hadoop-core-0.20.2.jar

Adding lib directory is best solution to fix this issue 

weblogic auto deployment script

Purpose : This script will copy the ear file from specific location and uninstall and install same ear file on the weblogic cluster . After the script will bounce the server .














## General Srcipt for Preprod App server , which have two Maneged servers 


## Config Variables

ADMINSERVERIP=10.192.168.1
ADMINSERVERPORT=7050

APPSERVER1IP=10.192.168.1
APPSERVER2IP=10.192.168.2

APPSERVER1PORT=8004
APPSERVER2PORT=9004

APPSERVER1NAME=appserver1
APPSERVER2NAME=appserver2

TARGET=mycluster

USERNAME=weblogic
PASSWORD=welcome1


APPLIB=/opt/app/bea/wls81sp6/user_projects/WLS1/applications
WLSDIR=/opt/app/bea/wls81sp6/user_projects/WLS1

SOURCE=/tmp
APPFILE=app.ear
APPILICATION=application

MAILLIST="yoyo.prod.support@company.com"
MAILSENDER="machineIP.host@@company.com"
MAILSUB="Deployment Status application  PREPROD  "


#### script starts from here

if ! id | grep oracle
then 
echo " please run as oracle " 
exit 90
fi

## /tmp/tt is the message file for the email content

mv /tmp/tt /tmp/tt.last
touch /tmp/tt


## sourcing evn file for jars required to run deployer utility


. $WLSDIR/bin/setDomainEnv.sh



if java weblogic.Admin -url $ADMINSERVERIP:$ADMINSERVERPORT  -username $USERNAME -password $PASSWORD GETSTATE | grep RUNNING
then
echo " Server is Running , Going for Deployment "
else
echo " Server is Not Running, Please  Start Manually "
exit 97
fi


cd $APPLIB
mv $APPFILE $APPFILE`date +%m%d%Y_%H%M `


cp  $SOURCE/$APPFILE .
chmod 755 $APPFILE

ls -ltr >> /tmp/tt

cksum $APPFILE >> /tmp/tt

if [ -f $APPFILE  ]
then
echo " file copied successfully "
else
echo " error in copying " >>/tmp/tt
echo " error in copying "
exit 96
fi


 

echo    >> /tmp/tt
echo " ^^^^^^^^^^^^^ Undeployment For  $APPFILE ^^^^^^^^^^ at  "` date +%H:%M:%S ` >> /tmp/tt
echo  >> /tmp/tt
if java weblogic.Deployer -url t3://$ADMINSERVERIP:$ADMINSERVERPORT  -username $USERNAME -password $PASSWORD -targets $TARGET -name $APPILICATION  -undeploy >> /tmp/tt
then
echo "File Undeployed Going for Deployment "
else
echo "Error in UnDeployment Exiting .... "
echo "Error in UnDeployment Exiting .... " >> /tmp/tt
mailx -s $MAILSUB $MAILLIST < /tmp/tt
exit 99
fi





echo >> /tmp/tt
echo "^^^^^^^^^^^^^ Deployment For $APPFILE ^^^^^^^^^^^^^ at  "` date +%H:%M:%S `  >> /tmp/tt
echo >> /tmp/tt
if  java weblogic.Deployer -url t3://$IP:$PORT  -username $USERNAME  -password $PASSWORD -targets EBPPCluster -name $TARGET  -deploy $APPLIB/$APPFILE1 >> /tmp/tt
then
echo "File deployed Going for Bouncing The Server "
echo "File deployed Going for Bouncing The Server ">> /tmp/tt
else
echo "Error in Deployment Exiting .... "
echo "Error in Deployment Exiting .... " >> /tmp/tt
mailx -s $MAILSUB $MAILLIST < /tmp/tt
exit 98
fi
echo >> /tmp/tt




echo " ^^^^^^^^^^^^ Killing  the server ^^^^^^^^^ at  "` date +%H:%M:%S ` >> /tmp/tt

echo >> /tmp/tt

java weblogic.Admin -url $APPSERVER1IP:$APPSERVER1PORT -username $USERNAME  -password $PASSWORD  FORCESHUTDOWN >> /tmp/tt 

java weblogic.Admin -url $APPSERVER2IP:$APPSERVER2PORT -username $USERNAME  -password $PASSWORD   FORCESHUTDOWN >> /tmp/tt 

java weblogic.Admin -url $ADMINSERVERIP:$ADMINSERVERPORT -username $USERNAME  -password $PASSWORD FORCESHUTDOWN >> /tmp/tt

echo >> /tmp/tt
sleep 5


# this is custom made script

cd $WLSDIR

./startAdmin1.sh

#./startWeblogic.sh &

count=0
while [ $count -le 120 ]

do
 if netstat -an | grep $PORT | grep LISTEN
  then
  echo " Server is UP "
  break
  else
  sleep 10
  count=`expr $count + 10`
 fi
done

if netstat -an | grep $PORT | grep LISTEN  >> /tmp/tt
 then
 echo " Server is UP after  $count sec " >>  /tmp/tt
 else
 echo " Server is not UP after $count sec Please check Manually " >> /tmp/tt
fi


## starting managed servers ,Node manager should be configured if not, start manually


java weblogic.Admin -url $ADMINSERVERIP:$ADMINSERVERPOR  -username $USERNAME  -password $PASSWORD START APPSERVER1NAME

sleep 60


java weblogic.Admin -url $ADMINSERVERIP:$ADMINSERVERPOR  -username $USERNAME  -password $PASSWORD START APPSERVER2NAME





mailx -r $MAILSENDER -s $MAILSUB $MAILLIST < /tmp/tt

How to get user details from weblogic access logs ?



Below shell script to get all logged in user , log in & log out time . You have to pass access.log as parameter


---------------------------------------------------------------------------------------------------------------------
if [ $# != 1 ]

then

echo "Usage : ./user.sh <access.log>"

exit 0

fi


file=$1

now=`date  +%s`

touch  /tmp/inactive /tmp/active


#### function to calculate the time out

timeout (){

   dt1=`echo $1 | sed 's:/:-:g; s/:/ /'`

              t1=`date --date="$dt1" +%s`

                let "tDiff=$now-$t1"
                let "hDiff=$tDiff/3600"
                

                if [ $hDiff -lt 1 ]
                  then
                     echo " $2 and last activity at `echo $lastact` " >> /tmp/active
                  else
                     echo " $2 and last activity at `echo $lastact`  " >> /tmp/inactive
                fi


         }




echo "unique  users logged in so far "

grep "adfAuthentication?login=true" $file | awk '{print $3}' |  sort -u  > /tmp/user.txt

more +2 /tmp/user.txt

echo "number of unique users loged in so far is   ` more +2 /tmp/user.txt | wc -l ` "

echo "##########################"

echo "users logged out so far "

grep "adfAuthentication?logout=true" $file | awk '{print $3}' |  sort -u  > /tmp/log_out.txt

more  /tmp/log_out.txt

echo "number of users  loged  out so far is   ` more /tmp/log_out.txt | wc -l ` "



more +2 /tmp/user.txt | while read LINE
  do

    login=`grep $LINE $file | grep "adfAuthentication?login=true" |  awk '{print $4}' | cut -f2 -d"[" `

    logout=`grep $LINE $file | grep "adfAuthentication?logout=true" | awk  '{print $4}' | cut -f2 -d"[" `



        if [ $logout ]

          then
             
              a=`echo $login | wc -w`
              b=`echo $logout | wc -w`
             
                 if [ $a -eq $b ]
                
                    then
                      
                        echo " $LINE login at `echo $login`  logout at  `echo $logout` " >> /tmp/inactive
                    
                    else

                        lastlogin=`echo $login | awk  '{print $NF}'`
                        lastout=`echo $logout| awk  '{print $NF}'`
                    
                        dli=`echo $lastlogin | sed 's:/:-:g; s/:/ /'`
                        tli=`date --date="$dli" +%s`

                        dlo=`echo $lastout | sed 's:/:-:g; s/:/ /'`
                        tlo=`date --date="$dlo" +%s`
                       
                              if [ $tlo -gt $tli ]

                                then
                           
                                   echo " $LINE logins at `echo $login `  logout at  `echo $logout` "  >> /tmp/inactive
                                else
                     
                                   lastact=`grep $LINE $file | tail -1 |  awk  '{print $4}' | cut -f2 -d"["`
                                   session="$LINE logins at `echo $login`  logout at  `echo $logout` "
                                   timeout "$lastact" "$session"

                              fi


                  fi

         else 
             
           session="$LINE login at `echo $login`"
           lastact=`grep $LINE $file | tail -1 |  awk  '{print $4}' | cut -f2 -d"["`
           timeout "$lastact"  "$session"
         fi


  done


echo "###############"
echo "Active Session "
more /tmp/active
echo "###############"
echo "Inactive Session "
more /tmp/inactive
echo "###############"
rm  /tmp/inactive /tmp/active
------------------------------------------------------------------------------------------------------------------------