OK, it is the time of the year for next year’s resolutions again, followings are what was happening in the last 12 months: Write more professional and personal blogs ( at least 4 each / month ) Need to start having the habit of Test Driven Development Make a website …
Before Christmas, I posted a blog about my trouble to write a Flume Collector Sink Decorator Plugin. After doing some research and continue digging the underlining issue, I finally get a solution to this, which makes me super happy. The issue I had was caused by error: This is due to …
In the last few days I have been trying to create a Flume sink decorator plugin to filter out any data that we consider as garbage (It is very disappointing that Flume doesn’t even support a very basic filtering capability). I have followed the steps on R&D Blog with slight modifications …
To create partitions on external tables in Hive is a bit tricky, as it is different from creating partitions on normal tables. To do it, you have to: create the external table first load data with partition info load other data in with different partition key. If you try to …
My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!