Filtering duplicate entries from Splunk events

Question

I am new to splunk and have got some splunk events as below

2019-06-26 23:45:36 INFO ID 123456 | Response Code 404 2019-06-26 23:55:36 INFO ID 123456 | Response Code 404 2019-06-26 23:23:36 INFO ID 258080 | Response Code 404

Is there way to filter out the first two events as they have the same ID 123456 and view them as one event? I tried something which I know is completely wrong, suggestions might be very useful on this.

index=myindex "Response Code 404" | rex field=ID max_match=2 "(?<MyID>\b(?:123456)\b)" | stats count by ID MyID | where count > 1

Answer 1

That's not completely wrong. It's one of the legitimate ways to remove duplicates. Here's another:

index=myindex "Response Code 404"  
| rex field=ID max_match=2 "(?<MyID>\b(?:123456)\b)" 
| dedup MyID

Using dedup is often preferred because it doesn't remove fields the way stats does.

Answer 2

I know it's late reply, but those really aren't duplicate events if the timestamps are different. I'd be more concerned about finding out what machine is sending an event twice at different times (and why than eliminating the results. Keep in mind that each event will go against your license, and while it may seem small, enough of them add up to a GB.