This article explains the workarounds to avoid the Hive query failure when processing snappy files generated by Hive and Flume under the same directory. The following are the steps to re-produce the issue: A Hive table (from_hive) with its data injected from Flume Create another table with same column structure (from_flume) Insert data …
Before Christmas, I posted a blog about my trouble to write a Flume Collector Sink Decorator Plugin. After doing some research and continue digging the underlining issue, I finally get a solution to this, which makes me super happy. The issue I had was caused by error: This is due to …
In the last few days I have been trying to create a Flume sink decorator plugin to filter out any data that we consider as garbage (It is very disappointing that Flume doesn’t even support a very basic filtering capability). I have followed the steps on R&D Blog with slight modifications …
Snappy is a compression/decompression library developed by Google. It aims for very high speeds and reasonable compression ( might be bigger than other standard compression algorithms but faster speed ). Snappy is shipped with Hadoop, unlike LZO compression which is excluded due to licensing issues. To enable Snappy in your …
To compile and install Hadoop’s LZO compression library on CentOS, following the steps below: Download hadoop LZO source from Kevin’s Hadoop LZO Project. If you are using Ant version of < 1.7, please download latest ant binary pacakge from Apache Ant, otherwise you will get the following error when compiling: …
My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!