This article explains the possible causes when HiveMetastore server (HMS) takes a long time to start up (more than 10 minutes).
Every time when Hive is restarted through Cloudera Manager (CM), it takes more than 10 minutes for Hive services to become green and users to be able to use beeline CLI.
One possible cause of the issue is:
- Sentry HDFS sync is enabled
- There are lots of tables or tables with lots of partitions (hundreds of thousands of partitions)
When the above two conditions are met, when HMS starts up, it will need to scan through all the tables and partitions in HMS database, and then sync with HDFS directories one by one. If there are too many tables or partitions, there will be a lot of HDFS directories that need to be synced, which will take some time.
To confirm whether this is the cause, simply disable the HDFS sync, restart Hive and see how long HMS takes to start up again. If the symptom disappears, then we have confirmed the cause.
The fix is to simply keep the number of tables and partitions per table down:
- If possible, drop the tables that you do not need
- If you need to keep the tables that have lots of partitions, try to merge those partitions if possible, by copying data into a new table with merged partitions
- If there are hundreds of thousands of partitions that you can not merge, then it is time to redesign your tables so that less partitions could be used
