In the Previous blog post, we went through how to use Apache Ambari to manage Big data stacks.
Now we will see how to use Ambari as resource management dashboard and how to share a resource among running application stack on the cluster.
For an example, I will take YARN application and will run multiple application over it.
YARN’s Capacity Scheduler is designed to run Hadoop applications in a shared, multi-tenant cluster while maximizing the throughput and the utilization of the cluster.
For an organization, it economically benefits when different units share a resource among them.
The fundamental unit of scheduling in YARN is a queue.
The capacity of each queue specifies the percentage of cluster resources that are available for applications submitted to the queue.
A Queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources.
- Go to Ambari Dashboard and click on YARN link, It will display all available resource on the cluster.
- click on the quick link and click the ResourceManager UI.
- At the Resource manager dashboard click on Scheduler Link.
CONFIGURING THE CAPACITY SCHEDULER
Specifying Which Version of Spark to Use
The default version for HDP 2.5.0 is Spark 1.6.2.
If more than one version of Spark is installed on a node, you can select which version of Spark runs your job.
To do this, set the SPARK_MAJOR_VERSION environment variable to the desired version before you launch the job.
For example, if Spark 1.6.2 and the Spark 2.1 technical preview are both installed on a node, and you want to run your job with Spark 2.1, set SPARK_MAJOR_VERSION to 2.1
For testing spark 2.0 You can run spark pi example.
cd /usr/hdp/current/spark2-client/ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples*.jar 10
You can check Yarn resource Manager UI in Ambari dashboard for running job status.