Impala Catalog Server keeps crashing

This articles explains one of the possibilities that could have caused catalogd to crash constantly.

The following are the symptoms when this issue happens:

• Impala Catalog server keeps crashing
• after restarting of Catalog server, query back to normal
• when impala is under heavy load, catalog server will crash again

One possible cause of the issue is memory consumption of the catalog server. After examining cluster information, we have identified the following facts within the cluster:

• The cluster has tens of millions of files in HDFS
• There are hundreds of thousands of partitions and close to 2 million partition parameters
• -load_catalog_in_background=true in the impala start up options

With so much metadata information, if loading of catalog in the background is set to true, when catalog server starts, it will try to load all the metadata information into memory. This can help with performance gain, however, when there are too many metadata information, it will become a bottleneck for memory usage. And when Impala becomes busy, weird things might happen.

So the solution in this situation here is to disable the loading of metadata information in the background for catalog server, so that information is only loaded when needed/requested by user.

To do this, following one of the steps below:

• If no Cloudera Manager is used, manually set -load_catalog_in_background=false on the Impala Daemon start up options.
• If Cloudera Manager is used, set “Load Catalog In Background” under Cloudera Manager > Impala > Configuration page, and then restart Impala Service.

That should help with the catalog daemon crash frequently issue.