Install Apache Oozie in Hadoop single node cluster

System Requirements for oozie 4.3.0 installation:

·        Unix box

·        Java JDK 1.6+

·        Maven 3.3.9

·        Hadoop 2.7.3

·        Pig 0.16.0

Assuming java and Hadoop are already installed, lets us install maven and pig

For Maven

I have Hadoop in the /srv path, so will download everything at this path

cd srv

$wget http://apache.mirror.amaze.com.au/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

tar xvfz apache-maven-3.3.9-bin.tar.gz

rm apache-maven-3.3.9-bin.tar.gz

For Pig

$wget http://apache.mirror.amaze.com.au/pig/latest/pig-0.16.0.tar.gz

tar xvfz pig-0.16.0.tar.gz

rm pig-0.16.0.tar.gz

For Oozie

wget http://apache.mirror.amaze.com.au/oozie/4.3.0/oozie-4.3.0.tar.gz

tar xvfz oozie-4.3.0.tar.gz

rm oozie-4.3.0.tar.gz 

Now set the environment variables

vi ~/.bashrc (or you can setup in the . ~/environment.sh)

export MAVEN_HOME=/srv/apache-maven-3.3.9

export PATH=$PATH:/srv/apache-maven-3.3.9/bin

Before running oozie build command, correct the paths in the action-conf/hive.xml file under oozie/conf directory, as per your hadoop implementation path

<property>

      <name>hadoop.bin.path</name>

      <value>/srv/hadoop-2.7.3/bin</value>

   </property>

   <property>

      <name>hadoop.config.dir</name>

      <value>/srv/hadoop-2.7.3/etc/hadoop</value>

   </property>

Go to oozie directory and run below command to build supported files

cd /srv/oozie-4.3.0

$bin/mkdistro.sh –DskipTests

If you don’t  pass –DskipTests, it runs some tests for 2hrs (if you allow tests, you can see more details about the tests it runs in the folder ~/oozie-4.3.0/core/target/test-data/oozietests, and the test reports are available in the folder ~/oozie-4.3.0/core/target/surefire-reports) do not interrupt until it finishes everything.

If error pops up, run the command $mvn clean And re run the same command again. If it still errors out, try running with bin/mkdistro.sh -DskipTests -X     x is for debug, but strangely it finished in success when ran.   

 

Oozie server setup

System Requirements

• Unix (tested in Linux and Mac OS X)
• Java 1.6+
• Hadoop
• Apache Hadoop (tested on 2.7.6)
• ExtJS library (optional, to enable Oozie webconsole)
• ExtJS 2.2

The Java 1.6+ bin directory should be in the command path.

Add below properties in the Hadoop core-site.xml

Standard properties

<!-- OOZIE -->

  <property>

    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>

    <value>[OOZIE_SERVER_HOSTNAME]</value>

  </property>

  <property>

    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>

    <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>

  </property>

In my case these properties will be 

<!-- OOZIE -->

  <property>

    <name>hadoop.proxyuser.hadoop.hosts</name>

    <value>localhost</value>

  </property>

  <property>

    <name>hadoop.proxyuser.hadoop.groups</name>

    <value>hadoop</value>

  </property>

Only for the lower versions: Now lets look at ExtJS, this is optional, do this only if you want web consle of oozie to monitor the jobs. You need to download extjs (a framework for oozie web console), only version 2.2 or 2.3 works for now, and can only be downloadable with url http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

Create a libext folder under the oozie directory

$cd /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0

$mkdir libext

If you have already download ExtJs zip file, move the file to this directory. If not downloaded, download with below command

$cd libext

$wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

Now copy Hadoop libraries and hcatalog libraries into libext folder of oozie.

copy all the jars from the paths

/srv/hadoop-2.7.3/share/hadoop/common/*.jar 

/srv/hadoop-2.7.3/share/hadoop/common/lib/*.jar

/srv/hadoop-2.7.3/share/hadoop/hdfs/*.jar

/srv/hadoop-2.7.3/share/hadoop/hdfs/lib/*.jar

/srv/hadoop-2.7.3/share/hadoop/mapreduce/*.jar 

/srv/hadoop-2.7.3/share/hadoop/mapreduce/lib/*.jar 

 /srv/hadoop-2.7.3/share/hadoop/yarn/*.jar  

/srv/hadoop-2.7.3/share/hadoop/yarn/lib/*.jar 

you can use the below unix command to copy jar files from sub directories

$cp **/*.jar /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/libext/

Now let us setup war file and sharelib with below commands.

$cd /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0

$ bin/oozie-setup.sh prepare-war 

Before creating sharelib, we have to copy all properties in /srv/hadoop-2.7.3/etc/hadoop/core-site.xml to 

the file /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/conf/hadoop-conf/core-site.xml

Also copy yarn and mapred site.xml files to oozie conf directory

 

cp /srv/hadoop-2.7.3/etc/hadoop/yarn-site.xml /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/conf/hadoop-conf
cp /srv/hadoop-2.7.3/etc/hadoop/mapred-site.xml /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/conf/hadoop-conf

And add below property to the /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/conf/oozie-site.xml 

 <property>
        <name>oozie.service.WorkflowAppService.system.libpath</name>
        <value>hdfs:///user/hadoop/share/lib/</value>
        <description>
            System library path to use for workflow applications.
            This path is added to workflow application if their job properties sets
            the property 'oozie.use.system.libpath' to true.
        </description>
    </property>
 

where hadoop is the your login user for hadoop. This way we dont face the "oozie could not locate sharelib" error.

Now run the below command

$bin/oozie-setup.sh  sharelib create -fs hdfs://localhost:54310 -locallib oozie-sharelib-4.3.0.tar.gz

Create database with below command

$bin/ooziedb.sh create -sqlfile oozie.sql -run

And run this command to start oozie daemon

$bin/oozied.sh start  

or to run as a process,

$bin/oozied.sh run

If you want to set up client node of oozie

$cd /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0

$tar xvfz oozie-client-4.3.0.tar.gz

$cd oozie-client-4.3.0/bin

$oozie

You can access the web console of oozie, go to the url

http://localhost:11000/oozie

or to check the status of oozie process, run the below command

$bin/oozie admin -oozie http://localhost:11000/oozie -status

to check available sharelibs

$bin/oozie admin -oozie http://localhost:11000/oozie -shareliblist

to see what sharelib is being used by oozie while the daemon is running,

$bin/oozie admin -sharelibupdate 

Installation of Oozie 4.3.0 is different to its lower versions, mainly prior versions will generate required hadoop libraries when build is complete. But in this version, we have to manually copy hadoop library jar files from hadoop distribution.

Also standard Oozie distribution comes with derby database which does not support multiple connections at the same time. Configuring Oozie with MySql is detailed in the below external references (by Apache) if required. 

Run the example oozie jobs and test the installation.

Firstly, navigate to the path /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/ and untar the examples file

$tar xvfz oozie-examples.tar.gz

$cd examples/apps

You see different example folders such as pig,hive,mapreduce etc.., i am taking mapreduce to explain below steps. In the job.properties files of each folder, change the name node port and job tracker port as highlighted below and add the parameter highlighted below. It should look like

From Hadoop 2.x, you must give resource manager port details to the Jobtracker property.

nameNode=hdfs://localhost:54310

jobTracker=localhost:8032

queueName=default

examplesRoot=examples

oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/hive

Then move the examples folder to hdfs

$cd /srv/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/

$Hadoop fs –put examples /user/Hadoop/examples

Submit the job, here I’m submitting oozie for mapreduce job

to run the oozie example

$bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

to check status of the job

$bin/oozie job -oozie http://localhost:11000/oozie -info 0000000-161215085944975-oozie-hado-W

to check the log of the job

$bin/oozie job -oozie http://localhost:11000/oozie -log 0000000-161215085944975-oozie-hado-W

Highlighted is the job id you need to replace. You can monitor the jobs in the web console.

 

Please leave the comment for any questions or clarifications. 

External references for further information

https://oozie.apache.org/docs/4.2.0/AG_Install.html

https://oozie.apache.org/docs/4.2.0/DG_QuickStart.html

https://oozie.apache.org/docs/4.2.0/ENG_Building.html

Possible errors while building oozie

 If you face any warnings with maven-clover2-plugin, navigate to the below link and see the latest version https://repo.maven.apache.org/maven2/com/atlassian/maven/plugins/maven-clover2-plugin/

and add the version tag in the pom.xml in the oozie directory for the property  maven-clover2-plugin