In the last few days I have been trying to create a Flume sink decorator plugin to filter out any data that we consider as garbage (It is very disappointing that Flume doesn’t even support a very basic filtering capability). I have followed the steps on R&D Blog with slight modifications to the HelloWorldSinkDecorator Java class. The code compiled but when I plugged it into Flume and enabled it, the collector simply died with no error message what so ever.
I had to stop working on this on Thursday afternoon as I had to join our company’s Xmas party lunch in the city. But with this outstanding task not finished before the break makes me very unhappy. Currently I still can’t find a solution and have to continue digging it when I go back to office next week.
Flume has so many problems since the first day I used it and it is not flexible enough to do lots of simply tasks. I am wondering whether we should be looking for alternatives.
Very interested to see how Flume NG will turn out.
