How to implement Custom Sink for a Embedded Flume Agent?

2
votes

I am building a spring boot standalone application that needs to consume messages from a remote server and write them to a Syslog Server. I am using Flume embedded agent to reliably log messages. But I am getting following error.

Caused by: org.apache.flume.FlumeException: Component type of com.security.flume.sink.SyslogSink is not in allowed types of [AVRO] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.checkAllowed(EmbeddedAgentConfiguration.java:308) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.validate(EmbeddedAgentConfiguration.java:182) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.configure(EmbeddedAgentConfiguration.java:199) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:151) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:98) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0]

As per the documentation Embedded Agent only supports Avro Sink. Does it mean we can not even write a custom sink?

I am fairly new to Flume. I really appreciate your help on this issue.

Thanks!

1
javaflume

1 Answers

0
votes

this means that for embedded agent you just can use AvroSink as a custom sink. However, there are some techniques you can use to collect data from an external source using custom components:

  • Create a custom client that is capable of establishing communication with Flume source that exists in prior, like AvroSources or SyslogTcpSource.

  • Create a custom flume source having direct communication with a preexisting client through an IPC or RPC protocol.

I hope this answer is not too late and can help you with something. Thanks for Reading!!