Splunk - No results, but search is correct

Question

EDIT:

Inspecting the search and confronting the logs of the failed search vs the successful search, it seems that the failed search job does not parse the search string. I have a log that says:

Generating search operator is null, short-circuiting data generation

On the contrary, on the successful search I have the log of the search parsing process. I want to repeat here that the searches have the SAME code.

Question:

A dashboard suddenly doesn't return nothing. I opened the search tool and it does not return nothing again. But, if I push the button to run the search again, it does return the expected data.

This happens only on my remote virtual Windows host. On my Linux private computer, same splunk version same indexes, all works.

I tried to reinstall splunk and also to clear var\run\splunk\dispatch folder, without success.

It seems the problem is with a subsearch that I do on many dashboards. It's a search with many joins. If I remove one of them, it works. But of course I need it all.

Indexes are OK, since I use them all also for populating inputs in the same page.

The subsearch is this one:

  [search index=A ID_A="$id_a$" ID_B="$id_b$" | 
  dedup ID_A | 
  join ID_B 
    [search index=B AND ID_C="$id_c$" | 
    dedup ID_B | 
    join ID_C 
      [search index=C AND ID_D="$id_d$" | 
      dedup ID_C | 
      table ID_C] | 
    table ID_B] | 
  table ID_A]

As you can see, it's data from a database, not log data. If I remove the join on ID_C, it works.

Any ideas?

Lucas, more data is needed, what is your "search with many joins" ? Are you using best practices like PostProcessing to avoid hitting the max number of events ceiling ? — Joao Figueiredo
Added the search to the answer. I'm not using PostProcessing for now, but the search have no problem in other dashboards running it directly using the search tool. — Marco Sulla

Joao Figueiredo Joao Figueiredo · Accepted Answer · 2015-02-17T15:08:46

Seems one of your subsearches is quietly performing harikiri after the number of seconds defined in limits.conf.
It's tempting to use JOIN for anyone coming from the SQL world but in splunk that function seldom is the answer as it's way, way slower. I'd suggest you'd use this as a starting point

(index=A "$id_a$" OR $id_b$ ID_A="$id_a$" ID_B="$id_b$") OR (index=B $id_c$ ID_C="$id_c$") OR (index=C "$id_d$" ID_D="$id_d$")
| dedup ID_A by ID_C, ID_B

The dedup clause will need some work as I can't do much more than imagining your underlining data

Reading this will greatly help you in the future, http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SQLtoSplunk

Splunk - No results, but search is correct

EDIT:

Question:

3 Answers