Oozie Workflow Failed with Error “stream exceeds limit”

Last week I was working with a customer trying to fix an issue that Oozie SSH action failed with “stream exceeds limit” error. This error does not appear in the Oozie launcher log, but rather in the Oozie server log. This means there is no problem launching the job, but failed when Oozie was trying to parse the output from the launcher. The full error message looks like below:

2018-06-13 02:24:38,879 WARN org.apache.oozie.servlet.CallbackServlet: 
SERVER[xxxx.xxxx.xxxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0557604-180517170833199-oozie-oozi-W] 
ACTION[0557604-180517170833199-oozie-oozi-W@ssh-4d02] 
URL[POST http://xxxx.xxxx.xxxx.com:11000/oozie/callback?id=0557604-180517170833199-oozie-oozi-W@ssh-4d02&status=ERROR] 
user error, stream exceeds limit [2,048]
java.lang.IllegalArgumentException: stream exceeds limit [2,048]
at org.apache.oozie.util.IOUtils.getReaderAsString(IOUtils.java:84)
at org.apache.oozie.servlet.CallbackServlet.doPost(CallbackServlet.java:117)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:612)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:748)

By checking the source code, I found out that it failed right here at CallbackServlet.java:

    /**
     * POST callback
     */
    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException,
            IOException {
        String queryString = request.getQueryString();
        CallbackService callbackService = Services.get().get(CallbackService.class);

        if (!callbackService.isValid(queryString)) {
            throw new XServletException(HttpServletResponse.SC_BAD_REQUEST, ErrorCode.E0402, queryString);
        }

        String actionId = callbackService.getActionId(queryString);
        if (actionId == null) {
            throw new XServletException(HttpServletResponse.SC_BAD_REQUEST, ErrorCode.E0402, queryString);
        }
        log = XLog.getLog(getClass());
        setLogInfo(actionId);
        log.debug("Received a CallbackServlet.doPost() with query string " + queryString);

        validateContentType(request, RestConstants.TEXT_CONTENT_TYPE);
        try {
            log.info(XLog.STD, "callback for action [{0}]", actionId);
            String data = IOUtils.getReaderAsString(request.getReader(), maxDataLen); // Failed here
.....

And maxDataLen is defined earlier in the class:

    @Override
    public void init() {
        maxDataLen = ConfigurationService.getInt(CONF_MAX_DATA_LEN); // maxDataLen defined here
    }

which is defined here:

    public final static String CONF_MAX_DATA_LEN = "oozie.servlet.CallbackServlet.max.data.len";

And then follow the path, we can find that oozie.servlet.CallbackServlet.max.data.len has default value of 2048

    
        oozie.servlet.CallbackServlet.max.data.len
        2048
        
            Max size in characters for the action completion data output.
        
    

This matches with the error we saw earlier:

java.lang.IllegalArgumentException: stream exceeds limit [2,048]

So from here, it is pretty clear that the config we need to change is oozie.servlet.CallbackServlet.max.data.len inside oozie-site.xml file. If you are using Cloudera Manager, please follow below steps: 1. Go to Cloudera Manager > Oozie > Configuration > “Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml” and enter:

    oozie.servlet.CallbackServlet.max.data.len
    8192

2. Save and restart Oozie. This will extend the CallbackServlet data to 8K, from default of 2K. It should be enough in most cases. If it still fails, then you would better find out from your application as to why it produces so much output, which Oozie will try to capture and store in its database. Hope above helps.