fail2ban does not match date pattern

Question

I am trying to debug my fail2ban filter and some weird error with respect to my custom datepattern and stumbled upon this documentation. According to that the output of the command fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "<HOST>" should show something like:

Date template hits:
|- [# of hits] date format
|  [1] Year-Month-Day Hour:Minute:Second

However this already does not work on my system since the output I get is:

Running tests
=============

Use   failregex line : <HOST>
Use      single line : 2013-09-19 02:46:12 1.2.3.4


Results
=======

Failregex: 0 total

Ignoreregex: 0 total

Date template hits:

Lines: 1 lines, 0 ignored, 0 matched, 1 missed
[processed in 0.07 sec]

|- Missed line(s):
|  2013-09-19 02:46:12 1.2.3.4

Why does fail2ban not even recognize the date as it should?

PS:

$ fail2ban-client -V
0.11.2
$ fail2ban-regex -V
0.11.2
$ uname -r
5.8.18-1-MANJARO

Answer 1

That won't work on my system either, which means that fail2ban apparently doesn't recognize that date format.

It will recognize that same date in different format though, for example:

fail2ban-regex "Sep 19 02:46:12 2013 1.2.3.4" "<HOST>"

Date template hits:
|- [# of hits] date format
|  [1] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?

And this is how you can force it to recognize your timestamp by specifying a custom date pattern:

fail2ban-regex -d '%Y-%m-%d %H:%M:%S' "2013-09-19 02:46:12 1.2.3.4" "<HOST>"

Date template hits:
|- [# of hits] date format
|  [1] Year-Month-Day 24hour:Minute:Second

Answer 2

The answer to your question is that you are correct, it won't work! They finally fixed it in 0.10 and improved in 0.11. Here is an example constructed by me using one of the authors of the fix answer to a question similar to yours:

Since 0.10 the handling around datepattern extraction is more precise (so fewer vulnerable), see #1583 for details. Shortly if no anchor specified at some side, it gets a word boundary automatically - in your case datepattern should match string to end of the word, but after the seconds there are still 3 digits 050 before time zone, so it does not match.

This example below worked fine in version 0.9.4:

# fail2ban-regex -v --datepattern=' ^%Y%m%d\s+%H%M%S' '20200313 122326050+1100 srv1 popserv 27511 31051 139647144740608 Note;AcctBadPswd(50/6) [email protected]:cmd=PASS <pswd>:fromhost=10.0.0.1' '(?:popserv).* (?:Note;AcctBadPswd).*fromhost=(?P<host>\S*).*'

Running tests
=============

Use      datepattern :  ^YearMonthDay\s+24hourMinuteSecond
Use   failregex line : (?:popserv).* (?:Note;AcctBadPswd).*fromhost=(?P<h...
Use      single line : 20200313 122326050+1100 srv1 popserv 27511 31051 1...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] (?:popserv).* (?:Note;AcctBadPswd).*fromhost=(?P<host>\S*).*
|      10.0.0.1  Fri Mar 13 12:23:26 2020
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1]  ^YearMonthDay\s+24hourMinuteSecond
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

On 0.11.1 it fails to match.

# fail2ban-regex -v --datepattern=' ^%Y%m%d\s+%H%M%S' '20200313 122326050+1100 srv1 popserv 27511 31051 139647144740608 Note;AcctBadPswd(50/6) [email protected]:cmd=PASS <pswd>:fromhost=10.0.0.1' '(?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*'

Running tests
=============

Use      datepattern : ^YearMonthDay\s+24hourMinuteSecond
Use   failregex line : (?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*
Use      single line : 20200313 122326050+1100 srv1 popserv 27511 31051 1...


Results
=======

Failregex: 0 total
|-  #) [# of hits] regular expression
|   1) [0] (?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [0] ^YearMonthDay\s+24hourMinuteSecond
`-

Lines: 1 lines, 0 ignored, 0 matched, 1 missed
[processed in 0.00 sec]

|- Missed line(s):
|  20200313 122326050+1100 srv1 popserv 27511 31051 139647144740608 Note;AcctBadPswd(50/6) [email protected]:cmd=PASS <pswd>:fromhost=10.0.0.1
`-

The problem is that the correct datepattern for the string: 20200313 122326050+1100 ... looks like ^%Y%m%d\s+%H%M%S%f?%z. Likely the 3 extra digits are milliseconds that will be now matched by %f?, where ? makes it optional (you can also replace it with \d+ or \d* if you don't need such millisecond precise time.

$ msg='20200313 122326050+1100 srv1 popserv 27511 31051 139647144740608 Note;AcctBadPswd(50/6) [email protected]:cmd=PASS <pswd>:fromhost=10.0.0.1'
$ fail2ban-regex -v --VD --datepattern='^%Y%m%d\s+%H%M%S%f?%z' "$msg" '(?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*'

Running tests
=============

Use      datepattern : ^YearMonthDay\s+24hourMinuteSecondMicroseconds?Zone offset
Use   failregex line : (?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*
Use      single line : 20200313 122326050+1100 srv1 popserv 27511 31051 1...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] (?:popserv).* (?:Note;AcctBadPswd).*fromhost=<ADDR>.*
|      10.0.0.1  Fri Mar 13 02:23:26 2020
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] ^YearMonthDay\s+24hourMinuteSecondMicroseconds?Zone offset
|      # weight: 1.000 (1.000), pattern: ^%Y%m%d\s+%H%M%S%f?%z
|      # regex:   ^((?P<Y>\d\d\d\d)(?P<m>1[0-2]|0[1-9]|[1-9])(?P<d>3[0-1]|[1-2]\d|0[1-9]|[1-9]| [1-9])\s+(?P<H>2[0-3]|[0-1]\d|\d)(?P<M>[0-5]\d|\d)(?P<S>6[0-1]|[0-5]\d|\d)(?P<f>[0-9]{1,6})?(?P<z>Z|UTC|GMT|[+-][01]\d(?::?\d{2})?))(?=\b|\W|$)
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

Please also note that the regular expression in the first example, doesn't contain enough anchors, therefore this makes it weak and can be triggered by the wrong entries to the log.

I constructed this answer using information from Sebres available here