Recently I have spent sometime to prepare some training contents for my colleague in the Indian team about how Cloudera Manager (CM) and CDH talk with each other to get the monitoring of services in CM working, and I think I might be useful if I can share such content so that the CM/CDH newbies can also be get the benefits.
I will break this into different posts so that I can keep each article short.
In the first part of this series, I will briefly talk about how Service Monitoring (SMON) and Host Monitoring (HMON) tools work with CDH and CM server itself.
Firstly, let’s have a look at the chart that I built below:
In order to simplify the chart, I have put all CDH services into one host and all managed by one CM Agent process, this can reduce the number of lines between each components in the chart.
Before we dive into the chart, let me explain a bit of what each component is:
CM Server:
It is the main component in Cloudera Manager, it has many responsibilities:
- Keeps runtime state of cluster, by keeping information in its backend database
- Keeps configurations of each component, also stored in CM’s backend database
- Provides Cloudera Manager API
- Provides Cloudera Manager web UI
- Manages Heart beat from Cloudera Manager Agent processes
- Communicate with varies Cloudera Management Services to provide necessary functionalities like:
- produce warnings
- produce alerts
- display charts
- triggering commands
- etc…
CM Agent:
It is available on every single host that is managed by Cloudera Manager, it is responsible to:
- start and stop processes (via supervisord)
- unpacking configurations and deploy on each host
- triggering installations
- monitoring hosts
Service Monitor (SMON)
It is responsible to:
- collect health and metric information about services and report back to user through CM’s web UI
- collect activity information from the YARN service, so that CM can display list of YARN jobs in CM’s interface
- collect activity information from Impala service, so that CM can display list of Impala queries in CM’s interface
Host Monitor (HMON)
It is responsible to:
- collects health and metric information about hosts, and pass those information to CM to be displayed in web UI
Now, let’s get back to the chart and see how the communication happen between each of them:
You can see that CM agent is installed on each host that runs CDH components, which are in term managed by Cloudera Manager. The agent keeps track of all the processes that are running on the host, like NameNode, Resource Manager, Job History Server etc, this is achieved via supervisord, which is a third party software that is integrated into Cloudera Manager. Discussion of supervisord falls outside the scope of this article, if you are interested, please review here.
CM agent needs to constantly (15 seconds by default) heartbeat to CM server, so that CM server knows the activities happened on this host. For example, the activities like start, stop or restart of services triggered on certain hosts will be sent to CM server as part of the heartbeat process, CM will update its backend database to record the new status, and in term sends instructions back to CM agent to tell what it should do next.
CM server will store all of its data into its backend database, including configurations for each components, status of each services and host. Most of CM/CDH users use MySQL, PostgreSQL or Oracle as CM’s backend database, as they are standard relational databases that are widely used.
CM agent also periodically sends host and service statuses to both Host Monitor and Service Monitor, so that the health status can be recorded in CM side and displayed on CM’s web UI. If there is any abnormal statuses detected, alerts/warnings will be triggered in CM and notify users.
This concludes the first part of the introduction to Cloudera Manager services. Stay tuned for the next part.
