Hive query failed with error: Killing the Job. mapResourceReqt: 1638 maxContainerCapability:1200″

Hive query failed with error: Killing the Job. mapResourceReqt: 1638 maxContainerCapability:1200″

This article explains how to fix the following error when running a hive query:
MAP capability required is more than the supported max container capability in the cluster. 
Killing the Job. mapResourceRequest:  maxContainerCapability:
rtaImage This error might not be obvious, however, this is caused by the following config not setup properly in YARN: mapreduce.map.memory.mb = 1638 yarn.scheduler.maximum-allocation-mb = 1200 yarn.nodemanager.resource.memory-mb = 1300 The solution here is to change the above mentioned property to have the following values: mapreduce.map.memory.mb < yarn.nodemanager.resource.memory-mb < yarn.scheduler.maximum-allocation-mb mapreduce.map.memory.mb is the amount of heap can be allocated to map task, which is inside a container. yarn.nodemanager.resource.memory-mb is the amount of heap to be allocated to the container that will hold either mappers or reducers. yarn.scheduler.maximum-allocation-mb is the amount of maximum memory can be allocated to container, which need to be higher than yarn.nodemanager.resource.memory-mb. Then the problem will be solved. Hope this helps.

Leave a Reply

Your email address will not be published.

My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!