When copying data across clusters using DistCp action through Oozie in a secured environment, due to a current Oozie bug, the job will fail with below error message:
java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "xxxxxxxxxx/xxx.xxx.xxx.xxx"; destination host is: "xxxxxxxxxx":8020; ... Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812) at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560) at org.apache.hadoop.ipc.Client.call(Client.java:1391) ... 43 more
This is a known issue and reported in the upstream JIRA of OOZIE-2900
This JIRA has been fixed in CDH6 and above. However, if you can not upgrade for the time being, you can add below configuration into Oozie workflow XML file to avoid the issue:
oozie.launcher.mapreduce.job.hdfs-servers hdfs://nameservice-cluster1,hdfs://nameservice-cluster2
Then re-run the Oozie job again to confirm the fix.
Hope this can help.
Hello,
Thank you for this post. I have fixed same issue.
Do you now how to fix this issue on Ooozie Server side?
Hi Viktor,
Thanks for posting comment on my blog. Can you please clarify on what issue on the Oozie Server side that you want to fix?
As mentioned OOZIE-2900 has been fixed since CDH6.0 and above.
Regards
Eric