Application log shows container “is running beyond physical memory limits” error

Application log shows container “is running beyond physical memory limits” error

This post explain what you need to do when you see container “is running beyond physical memory limits” error in the hive application log. Symptoms: Hive query failed with the following error in the application log:
2015-07-06 09:53:30,004 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: 
Diagnostics report from attempt_1435155934213_16721_m_002655_0: 
Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666] is running beyond physical memory limits. 
Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
Cause: There is not enough memory allocated to each container that will trigger the map tasks Solution: Find out the current memory for container and map task
SET mapreduce.map.memory.mb;
SET mapreduce.map.java.opts;
Then increase the heap size for both of them in Hive, the heap size for map task should slightly smaller than the container memory
SET mapreduce.map.memory.mb=2048;
SET mapreduce.map.java.opts=-Xmx1800M;
Then run the query again.

    2 Comments

    1. Chethan

      Hi, In our scenario we faced same issue while running oozie job, but with out setting any additional parameter the same job succeeded in the next run . any idea over here?

      1. Eric Lin

        Hi Chethan,

        Thanks for visiting my blog and post your question.

        It would be hard to understand what was going on without looking at why it failed in the first place. Have you checked the Oozie launcher log and try to find the error message?

        Cheers
        Eric

    Leave a Reply

    Your email address will not be published.

    My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!