Splunk rex query does not return desired result

Question

I am looking to search for error type in my spunk. A typical error log looks like this:

ERROR 2016/03/16 22:13:55 Program exited with error Calling service: Post http://hostname/v1.21/resource/create?name=/60b80cf9-ebc4-11e5-a9cb-3c4a92db9491-2: read unix @->/var/run/program.sock: use of closed network connection (Client.Timeout exceeded while awaiting headers)

Note that common part is "Program exited with error". I am looking to capture the part that follows this common part of the error message. I tried with a couple of rex expressions. Both returned different results. Importantly, neither captured the error type I have shown above. I am giving the one that worked better here.

* | rex "Program exited with error\s+(?<reason>.+)" |  top reason

An example of the log it matched- Unable to get program status, Get http://192.168.0.2:2774/program/v1/status: net/http: timeout awaiting response headers

However, it did not match log of the form-

initial ZK connection failed, stat /var/program/f47aae5c-ea42-11e5-8975-fc15b40f4cc4/srcheck/started: no such file or directory

Calling service: Post http://hostname/v1.21/resource/create?name=/60b80cf9-ebc4-11e5-a9cb-3c4a92db9491-2: read unix @->/var/run/program.sock: use of closed network connection (Client.Timeout exceeded while awaiting headers)

Could someone help me understand what's wrong with my rex expression and what the right one would be so I get all possible error types?

Answer 1

This recipe:

"ERROR.*Program exited with error.*:.*:.*:\s+(?<reason>.+)" 

will yield:

use of closed network connection (Client.Timeout exceeded while awaiting headers)

I don't have enough sample data to know if this will hold up or not. For example, I'm counting on exactly 3 colons, to get me to the interesting part. Also I don't know if you care about other things like the hostname, the fact that it's a Post, etc.. But based on your sample of 1, this answer should do the trick.