OK, it is the time of the year for next year’s resolutions again, followings are what was happening in the last 12 months: Write more professional and personal blogs ( at least 4 each / month ) Need to start having the habit of Test Driven Development Make a website …
Before Christmas, I posted a blog about my trouble to write a Flume Collector Sink Decorator Plugin. After doing some research and continue digging the underlining issue, I finally get a solution to this, which makes me super happy. The issue I had was caused by error: This is due to …
In the last few days I have been trying to create a Flume sink decorator plugin to filter out any data that we consider as garbage (It is very disappointing that Flume doesn’t even support a very basic filtering capability). I have followed the steps on R&D Blog with slight modifications …
To create partitions on external tables in Hive is a bit tricky, as it is different from creating partitions on normal tables. To do it, you have to: create the external table first load data with partition info load other data in with different partition key. If you try to …
Today I downloaded a zip file from one of our clients at work and tried to unzip the archive file, however, I got the following error: To solve this issue, I had to install p7zip. On Red Hat/CentOS, use yum: On Ubuntu, use apt-get: And then use command 7za to …
My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!