Last year, I wrote the first part of my Impala Query Profile series. I have been busy and not able to add the Part 2 of the series, apologies here. If you missed it, please check it out the last post here first: Impala Query Profile Explained – Part 1. …
As we know, Hadoop/HDFS/MapReduce/Impala is designed to store and process large amount of data, in terms of TBs or PBs. And we also know that having too many small files will hurt query performance, because NameNode needs to store millions of metadata to hold the information about files being stored …
After studying, working and living in Melbourne, Australia for almost 20 years, today marks my first day as an Australian Citizen. I arrived in Melbourne on 15th of January, 2000, as a high school student, at the age of 17, 3 months before my 18th birthday. After 4 years of …
When Distcp job runs and tries to copy data from HDFS to Amazon S3, Distcp job will try to buffer data to disk first until the output stream is closed via the close() method call. This is due to the nature of S3 object store, that data written to an …
My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!