How to iterate a JSON array of objects with jq and grab multiple variables from each object in each loop

Question

I need to grab variables from JSON properties.

The JSON array looks like this (GitHub API for repository tags), which I obtain from a curl request.

[
    {
        "name": "my-tag-name",
        "zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
        "tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
        "commit": {
            "sha": "commit-sha",
            "url": "https://api.github.com/repos/path-to-my-commit-sha"
        },
        "node_id": "node-id"
    },
    {
        "name": "another-tag-name",
        "zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
        "tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
        "commit": {
            "sha": "commit-sha",
            "url": "https://api.github.com/repos/path-to-my-commit-sha"
        },
        "node_id": "node-id"
    },
]

In my actual JSON there are 100s of objects like these.

While I loop each one of these I need to grab the name and the commit URL, then perform more operations with these two variables before I get to the next object and repeat.

I tried (with and without -r)

tags=$(curl -s -u "${GITHUB_USERNAME}:${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/path-to-my-repository/tags?per_page=100&page=${page}")
for row in $(jq -r '.[]' <<< "$tags"); do
    tag=$(jq -r '.name' <<< "$row")
    # I have also tried with the syntax:
    url=$(echo "${row}" | jq -r '.commit.url')
    # do stuff with $tag and $url...
done

But I get errors like:

parse error: Unfinished JSON term at EOF at line 2, column 0 jq: error (at :1): Cannot index string with string "name" } parse error: Unmatched '}' at line 1, column 1

And from the terminal output it appears that it is trying to parse $row in a strange way, trying to grab .name from every substring? Not sure.

I am assuming the output from $(jq '.[]' <<< "$tags") could be valid JSON, from which I could again use jq to grab the object properties I need, but maybe that is not the case? If I output ${row} it does look like valid JSON to me, and I tried pasting the results in a JSON validator, everything seems to check out...

How do I grab the ".name" and ".commit.url" for each of these object before I move onto the next one?

Thanks

jq is giving that error. Have you ensured that ${tags} has valid json? try printing it withe echo ${tags} — Rakesh Gupta
it is valid JSON, tried also with -r no change - pasted the output into a JSON validator, checks out — unfulvio

peak peak · Accepted Answer · 2021-06-17T05:50:08

It would be better to avoid calling jq more than once. Consider, for example:

while read -r name ; do
    read -r url
    echo "$name" "$url"
done < <( curl .... | jq -r '.[] | .name, .commit.url' )

where curl .... signifies the relevant invocation of curl.

How to iterate a JSON array of objects with jq and grab multiple variables from each object in each loop

1 Answers