I am having an issue that I am trying to resolve with IIS logs and Elasticsearch. What is happening is that my usernames in the IIS logs have a backslash () in them and not a forward slash (/). When Elasticsearch returns names it no longer has the \ which I would have hoped would have been escaped when entered. So when viewing the results in Elasticsearch or Kibana the username has no \ and the slash that is there is being treated as a regular expression. An example of this would be the username abcd\bob would be returned as abcdob.
I also believe that this issue would be the reason I am getting a _grokparsefailure tag added to each entry coming from IIS.
Any suggestions?
My NXLOG file getting the data:
## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> #<Extension w3c> #map iis log fields to Field Types # Module xm_csv # Fields $date, $time, $website, $serverip, $method, $url, $querystring, $port, $username, $clientip, $version, $useragent, $referer, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken # FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer # # #Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken # #FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer # Delimiter ' ' # QuoteChar '"' # EscapeControl FALSE # UndefValue - #</Extension> <Extension w3c> #map iis log fields to Field Types Module xm_csv Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer Delimiter ' ' </Extension> # Nxlog internal logs <Input internal> Module im_internal Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> # Windows Event Log <Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later Module im_msvistalog # Uncomment im_mseventlog for Windows XP/2000/2003 # Module im_mseventlog Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> <Input iis-logs> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' ReadFromLast TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ to_json (); \ } </Input> <Output out> Module om_tcp Host logs.{domain removed}.com Port 3515 </Output> <Output iis-out> Module om_tcp Host logs.{domain removed}.com Port 3516 </Output> <Route 1> Path internal, eventlog => out </Route> <Route 2> Path iis-logs => iis-out </Route>
My Logstash.conf file:
input { tcp { port => 5000 type => "syslog" } tcp { type => "eventlog" port => 3515 codec => json_lines } tcp { type => "iislog" port => 3516 codec => json_lines } } filter { if [type] == "syslog" { grok { match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] # add_field => [ "received_from", "%{@source_host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ] } if "_grokparsefailure" not in [tags] { mutate { replace => [ "@message", "%{syslog_message}" ] } } mutate { remove => [ "syslog_message", "syslog_timestamp" ] } kv { source => "@message" } } if [type] == "eventlog" { # Incoming Windows Event logs from nxlog # The EventReceivedTime field must contain only digits, or it is an invalid message # if [EventReceivedTime] !~ /\d+/ { drop { } } # grep { # match => [ "EventReceivedTime", "\d+" ] # } mutate { # Lowercase some values that are always in uppercase lowercase => [ "EventType", "FileName", "Hostname", "Severity" ] } mutate { # Set source to what the message says rename => [ "Hostname", "@source_host" ] } date { # Convert timestamp from integer in UTC match => [ "EventReceivedTime", "UNIX" ] } mutate { # Rename some fields into something more useful rename => [ "Message", "@message" ] rename => [ "Severity", "eventlog_severity" ] rename => [ "SeverityValue", "eventlog_severity_code" ] rename => [ "Channel", "eventlog_channel" ] rename => [ "SourceName", "eventlog_program" ] rename => [ "SourceModuleName", "nxlog_input" ] rename => [ "Category", "eventlog_category" ] rename => [ "EventID", "eventlog_id" ] rename => [ "RecordNumber", "eventlog_record_number" ] rename => [ "ProcessID", "eventlog_pid" ] } mutate { # Remove redundant fields remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ] } if [eventlog_id] == 4624 { mutate { add_tag => [ "ad-logon-success" ] } } if [eventlog_id] == 4634 { mutate { add_tag => [ "ad-logoff-success" ] } } if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 { mutate { add_tag => [ "ad-logon-failure" ] } } if [eventlog_id] == 4723 { mutate { add_tag => [ "ad-password-change" ] } } if [eventlog_id] == 4724 { mutate { add_tag => [ "ad-password-reset" ] } } if "ad-logon-success" in [tags] { metrics { add_tag => [ "drop", "metric", "ad-logon-success" ] meter => "ad-logon-success-metric" } } if "ad-logon-failure" in [tags] { metrics { add_tag => [ "drop", "metric", "ad-logon-failure" ] meter => "ad-logon-failure-metric" } } } if [type] == "iislog" { grok { # match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{IP:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"] match => ["message", "%{DATESTAMP:log_timestamp} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{NOTSPACE:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"] } useragent { source => "useragent" } #geoip { # source => "clientip" #} } metrics { meter => "events" add_tag => [ "drop", "metric", "events-metric" ] } } output { if "drop" not in [tags] { elasticsearch { host => "127.0.0.1" cluster => "logs" } # stdout { codec => rubydebug } } }
Sample IIS log entry:
2015-05-06 15:41:18 W3SVC2 WEB1 10.11.10.137 GET /main/ - 80 ABCD\smith 10.11.11.127 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+Touch;+.NET4.0E;+.NET4.0C;+Tablet+PC+2.0;+.NET+CLR+3.5.30729;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+InfoPath.3) cisession=a%3A4%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22959e25c7a1663350eeb85edb676de096%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2210.11.11.127%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A120%3A%22Mozilla%2F4.0+%28compatible%3B+MSIE+7.0%3B+Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+Touch%3B+.NET4.0E%3B+.NET4.0C%3B+Tablet+PC+2.0%3B+.NET+CL%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1430926793%3B%7D08a40eacaf8b6eba6102c7746c35c46497a6502a http://my.domain.com/main/scheduling my.domain.com 200 0 0 11314 1009 1458
UPDATE: I added a second nxlog iis input/output and instead of outputting the data to my logstash server I output to a flat file.
<Output iis2-out> Module om_file File 'C:\logs\logtest.txt' </Output>
I checked the output of this and noticed that the username has the backslash removed from the username prior to hitting the logstash server.
not_analyzed
string? Otherwise elasticsearch will keep trying to tokenize you usernames. – Felipe