Why there is a loss of data packets in Flume for syslogudp?

0
votes

I am seeing around 25% data loss using Flume source syslogudp.

Here is my configuration

a1.sources = r1

a1.sinks = k1

a1.channels = c1

Source

a1.sources.r1.type = syslogudp

a1.sources.r1.bind = 172.24.1.78

a1.sources.r1.port = 65535

sink

a1.sinks.k1.type = file_roll

a1.sinks.k1.sink.directory = /var/log/tmp

a1.sinks.k1.sink.rollInterval = 0

a1.sinks.k1.sink.batchSize = 20000

Channel

a1.channels.c1.type = memory

a1.channels.c1.capacity = 20000

a1.channels.c1.transactionCapacity = 20000

Bind

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

I have made changes in RHEL kernel parameters to solve the problem but no impact.

sysctl -w net.core.rmem_max=33554432

sysctl -w net.ipv4.udp_mem='262144 327680 393216'

sysctl -w net.core.netdev_max_backlog=2000

Any suggestions?

1
udpflume

1 Answers

0
votes

Maybe the problem is with UDP itself? Wikipedia says:

It (UDP) has no handshaking dialogues, and thus exposes any unreliability of the underlying network protocol to the user's program. As this is normally IP over unreliable media, there is no guarantee of delivery, ordering or duplicate protection.