Kibana: exporting and sorting embedded JSON using curl and jq

Question

Context:

I use Kibana (v5.5.1) on Ubuntu Xenial (16.04). I want to manage changes to Kibana objects (adding / removing / updating dashboards, visualizations, searches, etc.) in Git.

(Kibana's "snapshot" feature uses binary format, so it's not an option.)

I want to export Kibana objects as JSON and add/update them in Git.

I use curl and jq:

curl -s -XGET "http://localhost:9200/.kibana/dashboard/Metricbeat-Docker"| jq --sort-keys '.'

which is ok: I get diff-friendly sorted JSON that I can store in Git and later import back to Elastic.

Here's example output I get:

{
  "_id": "eslogs-*",
  "_index": ".kibana",
  "_source": {
    "fields": "[{\"name\":\"total_shards\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":false,\"aggregatable\":false},{\"name\":\"took_millis\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":false,\"aggregatable\":false},{\"name\":\"source\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false}]",
    "notExpandable": true,
    "timeFieldName": "@timestamp",
    "title": "eslogs-*"
  },
  "_type": "index-pattern",
  "_version": 1,
  "found": true
}

However, since field ordering in JSON is random, on every export I get too many false diffs caused by the field ordering. In order to make diff easy, I must sort 2 other things:

Sort the field ordering inside "_source" object.
Sort Embedded JSON text inside "_source.fields" by "name" field.

Regarding (2) I can use jq with -r option like this:

curl -s -XGET "http://localhost:9200/.kibana/index-pattern/eslogs-*" | jq  -r '._source.fields' | jq -c '. |= sort_by(.name)'

which gives me a nice sorted JSON.

I tried to get that sorted embedded JSON text into a shell variable and later to "inject" it to jq using --arg, but then I got "Argument too long" error.

So my question is:

How to combine all these requirements into a single command line script which do all the following:

Iterate over all Kibana objects
Sorts the field order in each JSON object
Sort the embedded JSON text inside "_source.fields"

Do you have any idea which is simple enough, readable and elegant?

Thanks.

Jeff Mercado Jeff Mercado · Accepted Answer · 2017-08-27T17:42:10

The thing about the -S option is that it only affects the output, it doesn't have any influence on the objects as they are being processed. Since the inner json string is not an object, jq cannot sort the keys in those objects within. If you wanted jq to sort them via the -S option, you would have to output it as objects first, then use another invocation to convert back to json.

$ jq -S '._source.fields |= (fromjson | sort_by(.name))' input.json | \
jq '._source.fields |= tojson'

That however is far from ideal. Instead, you could take advantage of the fact that jq preserves insertion order of object keys by default. So you could define a function that would take an object and recreate it with sorted keys. Then apply it to objects you want sorted. Then apply this recursively on all objects in your tree.

def sort_object: (objects | . as $in | reduce keys[] as $k ({}; .[$k] = $in[$k])) // .;
def sort_object_recursive: walk((objects | sort_object) // .);
._source.fields |= (fromjson | sort_by(.name)) | sort_object_recursive | ._source.fields |= tojson

This yields the following output:

{
  "_id": "eslogs-*",
  "_index": ".kibana",
  "_source": {
    "fields": "[{\"aggregatable\":false,\"analyzed\":true,\"count\":0,\"doc_values\":false,\"indexed\":true,\"name\":\"source\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"took_millis\",\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"total_shards\",\"scripted\":false,\"searchable\":false,\"type\":\"number\"}]",
    "notExpandable": true,
    "timeFieldName": "@timestamp",
    "title": "eslogs-*"
  },
  "_type": "index-pattern",
  "_version": 1,
  "found": true
}

Just be sure to include the definition of walk/1 in your ~/.jq file.

Kibana: exporting and sorting embedded JSON using curl and jq

1 Answers