Pages

Wednesday, July 6, 2016

How to install and configure kerbores for ssh authentication

How to install and configure kerbores for ssh authentication

Time should be in synchoroes between server and clients . Ticket is time based .
All variables are case senstive


In server :

Server Hostname : master



1. Install krb packages


yum install krb5-server krb5-libs krb5-auth-dialog
yum install krb5-pkinit-openssl


2. Edit kdc.conf & kadm5.acl files

Location : /var/kerberos/krb5kdc/

Currently only kdc.conf & kadm5.acl files located here

cd /var/kerberos/krb5kdc/

vi kdc.conf


Change EXAMPLE.COM to your relam

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
COMPANY.COM= {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}





3.


vi kadm5.acl


Edit */admin@COMPANY.COM *


*/admin@COMPANY.COM *


4. Edit krb5.conf
Location : /etc/



vi /etc/krb5.conf



[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = COMPANY.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true

[realms]
COMPANY.COM = {
kdc = master.COMPANY.COM
admin_server = master.COMPANY.COM
}

[domain_realm]
.COMPANY.COM = COMPANY.COM
COMPANY.COM = COMPANY.COM




5. Create Kerbores Database


cd /var/kerberos/krb5kdc/

kdb5_util create -s -r COMPANY.COM

Provide username/password

Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'COMPANY.COM',
master key name 'K/M@COMPANY.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
kdb5_util: File exists while creating database '/var/kerberos/krb5kdc/principal'



This action will create new files

principal ,principal.kadm5 etc

6. start krb5kdc service

service krb5kdc start


Starting Kerberos 5 KDC: [ OK ]

7. Access kadmin.local

kadmin.local

Authenticating as principal root/admin@COMPANY.COM with password.

kadmin.local:

8. In the kadmin.local add principal
a.
kadmin.local: listprincs

This will list it

b.
kadmin.local: addprinc root/admin

This will add principal root to admin group

Add Host Names



c.
kadmin.local: addprinc -randkey hosts/master.COMPANY.COM


d.
kadmin.local: ktadd hosts/master.COMPANY.COM

Add oracle user

e.

kadmin.local: addprinc oracle



9 . To get user token during login , then kerbores authentication should be enabled



authconfig-tui

OR


setup
(Then select authentication option)




Select Kerberos option



























Next and provide the kerbores server OPTION ( by default it will load from /etc/krb5. File


10. Verify login action (Existing user oracle)




a.
ssh -l oracle master

b.
kinit

Password for oracle@COMPANY.COM:

c.

klist

Ticket cache: FILE:/tmp/krb5cc_500
Default principal: oracle@COMPANY.COM

Valid starting Expires Service principal
07/06/16 23:35:47 07/07/16 23:35:47 krbtgt/COMPANY.COM@COMPANY.COM
renew until 07/06/16 23:35:47




11. Add the user with out password



a.

adduser sterin

b. Add in Kerberos DB


kadmin.local


kadmin.local: addprinc sterin

Provide username and password

c. Login using ssh


ssh -l sterin master

Now password is not stored in /etc/passwd of linux server, it will take from Kerberos DB

sterin@master's password:
[sterin@master ~]$ klist


Ticket cache: FILE:/tmp/krb5cc_501_Oc6cOn
Default principal: sterin@COMPANY.COM

Valid starting Expires Service principal
07/07/16 00:45:54 07/08/16 00:45:54 krbtgt/COMPANY.COM@COMPANY.COM
renew until 07/07/16 00:45:54

12. Check local user not in Kerberos DB


adduser notinkb
passwd notinkb

Changing password for user notinkb.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.


ssh -l notinkb master



kinit


kinit: Client not found in Kerberos database while getting initial credentials




To enable service in every restart 






 /sbin/chkconfig --add  krb5kdc
 /sbin/chkconfig krb5kdc on







In client

hostname client1.company.com




yum install krb5-workstation krb5-libs krb5-auth-dialog


2. Change the authentication ,by running setup command

Select Kerberos

























provide server details








3. Now try with user

4.klist will show the ticker details



Monday, July 4, 2016

How to install Ambari on Linux ?

How to install Ambari on OEL 6


1. hostname -f should show correct hostname , instead of localhost.localdomain

hostname -f

masterserver.xx.com

Hint : /etc/hosts should have correct entry and hostname details should be top

2. Get yum repo in /etc/yum.repo


cd /etc/yum.repo
wget http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo

3.yum update


yum update

(This action is critical because openssl lib should be updated )


4. Install using yum



yum install ambari-server


5. run ambari-setup

yum install ambari-server setup

Accept all default values .



Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y



Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...



Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 1

Installing JDK to /usr/jdk64/
Successfully installed JDK to /usr/jdk64/
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jce_policy-8.zip

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip
Installing JCE policy...
Completing setup...
Configuring database...



Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
==============================================================================
Enter choice (1): 1
Database name (ambari):
Postgres schema (ambari):
Username (ambari):
Enter Database Password (bigdata):
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Connecting to local database...done.
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Extracting system views...
.ambari-admin-2.2.2.0.460.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


6. Set http_proxy if needed in your network


vi /var/lib/ambari-server/ambari-env.sh




-Dhttp.proxyHost=www-myproxy.idc.com -Dhttp.proxyPort=80

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=www-proxy.us.oracle.com -Dhttp.proxyPort=80 '

7 . Verify the hostname in /etc/ambari-agent/conf/ambari-agent.ini


vi /etc/ambari-agent/conf/ambari-agent.ini


hostname=masterserver.xx.com
url_port=8440
secured_url_port=8441

This should match with step 1 .

8. Start ambari server

ambari-server start


9 . Access the URL


http://masterserver.xx.com:8080

Username : admin
Pass : admin

10. Provide the cluster Name

11. Provide the hostnames ( hostname -f )

12. Provide server id_rsa

13. Make sure that password less login works between master and all nodes

and permission to .ssh folder of all should be 600

14 . Select the component and proceed with installation



Wednesday, June 29, 2016

MapReduce Two Values for One key example

MapReduce  Multiple values for a single key

MapReduce  Joint example






In this example , creating MapReduce code for doing activity from hortonworsks

http://hortonworks.com/hadoop-tutorial/how-to-process-data-with-apache-pig/

It need to map , one key to two values .

Year as key and PlayerID & Runs as Value






This also contains csv creator in any size and standalone java code to do the same activity



Details of the files :


BattingExample.java : MapReduce Driver Class

BattingMapper.java : MapReduce Mapper Class

BattingReducer.java : MapReduce Reducer Class


Batting.jar : MapReduce Jar


hadoop jar ./Batting.jar BattingExample <InputCSVfile> <OutputFolder>




StandAlone.java : it used for run same Mapper Reduce logic in stand alone mode


java StandAlone <inputCSV> <outputFIle>


Batting.csv : it contains data of players

CsvCreator.java : Used to create similar csv file with any size in similar format



java CsvCreator <NumberOfPlayers> <outputCSVfile>

2000 players will create 1 MB file


Sunday, June 19, 2016

How to protect webUI port of namenode ?

How to protect webUI port 50070  of namenode ?

By default webUI port of namenode running on 50070 is not protected and details of HDFS and file system in read only mode are open to all , by accessing http://<namenodeServer>:50070

All hadoop daemons use an embedded Jetty web container to host JSP for webui.



Version used in the example : apache 2.7.2

1. Go to <hadoop_home>/ share/hadoop/hdfs/webapps/hdfs/WEB-INF

2. edit web.xml

From
<web-app version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee">
</web-app>


<web-app version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee">
</web-app>

To

<web-app version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee">
<security-constraint>
<web-resource-collection>
<web-resource-name>Protected</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>explorerRelam</realm-name>
</login-config>
</web-app>


3. Create new file : jetty-web.xml
CopyPaste:

<Configure class="org.mortbay.jetty.webapp.WebAppContext">
<Get name="securityHandler">
<Set name="userRealm">
<New class="org.mortbay.jetty.security.HashUserRealm">
<Set name="name">explorerRelam</Set>
<Set name="config">
<SystemProperty name="hadoop.home.dir"/>/jetty/etc/realm.properties
</Set>
</New>
</Set>
</Get>
</Configure>



4. Create new file <hadoop_home>/jetty/etc/realm.properties
(folder jetty/etc should be created )

format :

Username: password,group


tushar: welcome1,admin

5. Access http://IP:50070



























6. If only explorer need to protect use , in step 2


<url-pattern>/explorer.html/*</url-pattern>




Thursday, June 16, 2016

How to overwrite or update a file in hadoop HDFS ?

Using put or  copyFromLocal   wont able to update a file in HDFS . it will show below error


[hduser@localhost SampleData]$ hadoop fs -put books.csv  /yesB
put: Target /yesB/books.csv already exists


[hduser@localhost SampleData]$ hadoop fs -copyFromLocal   books.csv  /yesB
copyFromLocal: Target /yesB/books.csv already exists



To overcome this issue , distcp can be used 


 hadoop distcp -update  file://<source>  hdfs://<IP:PORT>/<targetlocation>

Example :

 hadoop distcp -update   file:///home/hduser/pigSample/labfiles/SampleData/books.csv hdfs://11.181.37.158:9000/yesB


-overwrite can be used . But using -update is better because it copy and do mapreduce only when there is difference in source and target 






Wednesday, May 25, 2016

java.lang.ClassNotFoundException and No FileSystem for scheme: hdfs

java.lang.ClassNotFoundException and No FileSystem for scheme: hdfs  exception while connecting to hadoop from application server



Stand alone code to connect to hadoop works fine But when the same code put it in application server ( weblogic ) , it fails . Even though all the jars bundled in ear file


Error 1 :

java.io.IOException: No FileSystem for scheme: hdfs                                                                        
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1600)                                          
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:69)                                                  
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1637)                                        
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1619)                                                
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:255)                                                        
        at connector.HadoopService.hadoopHandler(HadoopService.java:62)                                                    
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                    
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)                                  
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingM


Fix :
Edit the code  like this

                      Configuration conf = new Configuration();
               

              conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
             conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());



        FileSystem hdfs = FileSystem.get(new URI("hdfs://IP:9000"),conf);




Error 2 :


java.lang.ClassNotFoundException: Class  org.apache.hadoop.hdfs.DistributedFileSystem
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2290)
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2303)
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)




Fix : add all the jars required specifically in <domain>/lib directory

For hadoop 2.7.2  : required jars  ( all are located at common/lib folder )


  1. commons-io-2.4.jar       
  2.   guava-11.0.2.jar    
  3.    hadoop-common-2.7.2.jar 
  4.  htrace-core-3.1.0-incubating.jar 
  5.  protobuf-java-2.5.0.jar 
  6.  slf4j-api-1.7.10.jar
  7. commons-logging-1.1.3.jar 
  8.  hadoop-auth-2.7.2.jar 
  9.  hadoop-hdfs-2.7.2.jar  
  10.   log4j-1.2.17.jar   
For coludera distribution 

1. log4j-1.2.17.jar


2. commons-logging-1.0.4.jar

3. guava-r09-jarjar.jar


4. hadoop-core-0.20.2.jar


Adding lib directory is best solution to fix this issue