HDFS to S3 BDR Job Filled Up Disk Space
When Distcp job runs and tries to copy data from HDFS to Amazon S3, Distcp job will try to buffer data to disk first until the output stream is closed via the close() method call. This is due to the nature of S3 object store, that data written to an …
