Unable to form the JOLT schema to transform JSON in NiFi

Question

I am trying to use the jolt JSON to JSON transformation in Apache NiFi. I want to transform one JSON into another format.

Here is my original JSON:

{
  "total_rows": 5884,
  "offset": 0,
  "rows": [
    {
      "id": "03888c0ab40c32451a018be6b409eba3",
      "key": "03888c0ab40c32451a018be6b409eba3",
      "value": {
        "rev": "1-d5cc089dd8682422962ccab4f24bd21b"
      },
      "doc": {
        "_id": "03888c0ab40c32451a018be6b409eba3",
        "_rev": "1-d5cc089dd8682422962ccab4f24bd21b",
        "topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
        "payload": {
          "temperature": 36
        },
        "deviceId": "1234",
        "deviceType": "home-iot",
        "eventType": "temp",
        "format": "json"
      }
    },
    {
      "id": "03888c0ab40c32451a018be6b409f163",
      "key": "03888c0ab40c32451a018be6b409f163",
      "value": {
        "rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5"
      },
      "doc": {
        "_id": "03888c0ab40c32451a018be6b409f163",
        "_rev": "1-dee82cbb1b5ffa8a5e974135eb6340c5",
        "topic": "iot-2/type/home-iot/id/1234/evt/temp/fmt/json",
        "payload": {
          "temperature": 22
        },
        "deviceId": "1234",
        "deviceType": "home-iot",
        "eventType": "temp",
        "format": "json"
      }
    }
  ]
}

I want this to be transformed in the following JSON:

[
   {
      "temperature":36,
      "deviceId":"1234",
      "deviceType":"home-iot",
      "eventType":"temp"
   },
   {
      "temperature":22,
      "deviceId":"1234",
      "deviceType":"home-iot",
      "eventType":"temp"
   }
]

This is what my spec looks like:

[
  {
    "operation": "shift",
    "spec": {
      "rows": {
        "*": {
          "doc": {
            "deviceId": "[&1].deviceId",
            "deviceType": "[&1].deviceType",
            "eventType": "[&1].eventType",
            "payload": {
              "*": "[&1]"
            }
          }
        }
      }
    }
  }
]

I keep getting a null response. I am new to this and the documentation is not very easy to comprehend. Can somebody please help?

mattyb mattyb · Accepted Answer · 2019-04-09T13:34:29

Because you are "down" one more level after the array index, by the time you get to deviceId you are 2 levels away from the index. Replace all the &1s with &2 except for payload. In that case you are another level "down" so you'll want to use &3 for the index. You also need to take whatever is matched by the * (temperature, e.g.) and set the outgoing field name to the same thing, by using & after the array index. Here's the resulting spec:

[
  {
    "operation": "shift",
    "spec": {
      "rows": {
        "*": {
          "doc": {
            "deviceId": "[&2].deviceId",
            "deviceType": "[&2].deviceType",
            "eventType": "[&2].eventType",
            "payload": {
              "*": "[&3].&"
            }
          }
        }
      }
    }
  }
]

Unable to form the JOLT schema to transform JSON in NiFi

1 Answers