2018-06-13 02:24:38,879 WARN org.apache.oozie.servlet.CallbackServlet: SERVER[xxxx.xxxx.xxxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0557604-180517170833199-oozie-oozi-W] ACTION[0557604-180517170833199-oozie-oozi-W@ssh-4d02] URL[POST http://xxxx.xxxx.xxxx.com:11000/oozie/callback?id=0557604-180517170833199-oozie-oozi-W@ssh-4d02&status=ERROR] user error, stream exceeds limit [2,048] java.lang.IllegalArgumentException: stream exceeds limit [2,048] at org.apache.oozie.util.IOUtils.getReaderAsString(IOUtils.java:84) at org.apache.oozie.servlet.CallbackServlet.doPost(CallbackServlet.java:117) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:612) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503) at java.lang.Thread.run(Thread.java:748)By checking the source code, I found out that it failed right here at CallbackServlet.java:
/** * POST callback */ @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String queryString = request.getQueryString(); CallbackService callbackService = Services.get().get(CallbackService.class); if (!callbackService.isValid(queryString)) { throw new XServletException(HttpServletResponse.SC_BAD_REQUEST, ErrorCode.E0402, queryString); } String actionId = callbackService.getActionId(queryString); if (actionId == null) { throw new XServletException(HttpServletResponse.SC_BAD_REQUEST, ErrorCode.E0402, queryString); } log = XLog.getLog(getClass()); setLogInfo(actionId); log.debug("Received a CallbackServlet.doPost() with query string " + queryString); validateContentType(request, RestConstants.TEXT_CONTENT_TYPE); try { log.info(XLog.STD, "callback for action [{0}]", actionId); String data = IOUtils.getReaderAsString(request.getReader(), maxDataLen); // Failed here .....And maxDataLen is defined earlier in the class:
@Override public void init() { maxDataLen = ConfigurationService.getInt(CONF_MAX_DATA_LEN); // maxDataLen defined here }which is defined here:
public final static String CONF_MAX_DATA_LEN = "oozie.servlet.CallbackServlet.max.data.len";And then follow the path, we can find that oozie.servlet.CallbackServlet.max.data.len has default value of 2048
This matches with the error we saw earlier:oozie.servlet.CallbackServlet.max.data.len 2048 Max size in characters for the action completion data output.
java.lang.IllegalArgumentException: stream exceeds limit [2,048]So from here, it is pretty clear that the config we need to change is oozie.servlet.CallbackServlet.max.data.len inside oozie-site.xml file. If you are using Cloudera Manager, please follow below steps: 1. Go to Cloudera Manager > Oozie > Configuration > “Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml” and enter:
2. Save and restart Oozie. This will extend the CallbackServlet data to 8K, from default of 2K. It should be enough in most cases. If it still fails, then you would better find out from your application as to why it produces so much output, which Oozie will try to capture and store in its database. Hope above helps.oozie.servlet.CallbackServlet.max.data.len 8192
Tkx for this topic ! 🙂