How to prettify JSON documents at the terminal

Print Friendly, PDF & Email

Nowadays, many users and application developers know that using RESTful APIs is the most common and useful to grab remotely published data. It’s often to look at any RESTful API before data are integrated into applications. This testing can be performed through various tools. Depending on each user case, they could be shell console windows via curl command, API consumer clients like Postman or direct applications. It is not difficult to find how to do it, for example, this and that question. In this post, we want to summarise some ways to make the JSON output of the curl terminal command prettier so that we can look into it more easily.

Why do we need to prettify JSON outputs?

Let’s take a look at an example below.

$ cat revisions.json
{"vcsId":"aaa","publishedRevs":[2,1],"privateRevs":[],"submissionId":"MODEL6624243033"}

Does it look ugly? Is it easy to see? How about the other version below?

$ python3 -m json.tool revisions.json
{
    "vcsId": "aaa",
    "publishedRevs": [
        2,
        1
    ],
    "privateRevs": [],
    "submissionId": "MODEL6624243033"
}

Frankly speaking, we prefer the latter to the former.

Using the combination of Linux built-in commands

If the installation of required Python or Node packages and modules is a challenge because of lacking permissions, we suggest using Linux built-in commands and grouping them into a bash alias for our convenience.

json_prettier() {
   grep -Eo '"[^"]*" *(: *([0-9]*|"[^"]*")[^{}\["]*|,)?|[^"\]\[\}\{]*|\{|\},?|\[|\],?|[0-9 ]*,?'|awk'{if ($0 ~ /^[}\]]/ ) offset-=4; printf "%*c%s\n", offset, " ", $0; if ($0 ~ /^[{\[]/) offset+=4}'
}

For example:

~/tmp$ cat revisions.json | json_prettier
 {
    "vcsId":"aaa",
    "publishedRevs":
    [
        2,
        1
    ],
    "privateRevs":
    [
    ],
    "submissionId":"MODEL6624243033"
 }

Using python command

From Python 2.6+ or 3, we can use the json.tool module:

echo '{"forename": "Tom", "familyname": "Nguyen"}' | python -m json.tool

or, if the JSON document is in a file, we can do:

python -m json.tool users.json

If the JSON is from an internet source such as a RESTful API, we can use

curl http://my_url/ | python -m json.tool

For convenience in all of these cases we can make an alias:

alias prettyjson='python -m json.tool'

For even more convenience with a bit more typing to get it ready:

// for an inline JSON string
prettyjson_s() { 
  echo "$1" | python -m json.tool 
} 

// for a JSON file
prettyjson_f() { 
  python -m json.tool "$1" 
} 

// for RESTful API endpoints
prettyjson_w() { 
  curl "$1" | python -m json.tool 
}

for all the above cases. We are recommended to put those aliases in .bashrc or .zshrc and it will be available every time in shell. Invoke it like prettyjson_s '{"forename": "Tom", "familyname": "Nguyen"}'.

It’s helpful to notice that as in Python 3.5+, the JSON object is no longer sorted by default. To sort, add the --sort-keys flag to the end. I.e. ... | python -m json.tool --sort-keys.

Another useful option might be --no-ensure-ascii which disable the escaping of non-ASCII characters (new in version 3.9).

Using jq utility

It’s very simple to use and it works great! It can handle very large JSON structures, including streams. We recommend you go through their tutorials here.

Usage examples:

 ~/tmp$ cat revisions.json| jq
{
  "vcsId": "aaa",
  "publishedRevs": [
    2,
    1
  ],
  "privateRevs": [],
  "submissionId": "MODEL6624243033"
}

Take another example with a REST API URL below.

~/tmp$ curl -s https://wwwdev.ebi.ac.uk/biomodels/BIOMD0000000100\?format\=json | jq
{
  "name": "Rozi2003_GlycogenPhosphorylase_Activation",
  ...
  ...
  ...
  "publication": {
    "journal": "Biophysical chemistry",
    "title": "A theoretical study of effects of cytosolic Ca2+ oscillations on activation of glycogen phosphorylase.",
    "year": 2003,
    "month": "12",
    "volume": "106",
    "pages": "193-202",
    "link": "http://identifiers.org/pubmed/14556891",
    "authors": [
      {
        "name": "Rozi A"
      },
      {
        "name": "Jia Y"
      }
    ]
  },
  ...
  ...
  ...
  "history": {
    "revisions": [
      {
        "version": 1,
        "submitted": 1174604088000,
        "submitter": "Harish Dharuri",
        "comment": "Original import of Rozi2003_GlycogenPhosphorylase_Activation"
      },
      {
        "version": 2,
        "submitted": 1400953919000,
        "submitter": "Harish Dharuri",
        "comment": "Current version of Rozi2003_GlycogenPhosphorylase_Activation"
      }
    ]
  },
  "firstPublished": 1174559312000,
  "submissionId": "MODEL4589754842",
  "publicationId": "BIOMD0000000100"
}

Using http command

Let’s take a look at the http command which is installed from HTTPie: a modern, user-friendly command-line HTTP client for the API era.

For example:

~/tmp$ http https://wwwdev.ebi.ac.uk/biomodels/BIOMD0000000100\?format\=json
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json;charset=utf-8
Date: Sun, 24 Mar 2024 11:46:52 GMT
Server: nginx/1.19.1
Set-Cookie: biomodels-session=1711280812.859.16430.675266; Expires=Sun, 24-Mar-24 12:46:51 GMT; Max-Age=3600; Path=/biomodels; HttpOnly
Set-Cookie: SESSION=14fc2bf8-849a-4df8-8673-f66a60ac14de; Path=/biomodels/; HttpOnly
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Cache-Info: caching

{
    "description": "<notes xmlns=\"http://www.sbml.org/sbml/level2\">\r\n      <body xmlns=\"http://www.w3.org/1999/xhtml\">\r\n        <p>The model reproduces the temporal evolution of Glycogen phosphorylase for a vale of Vm5=30 as depicted in Fig 1a of the paper. The model makes use of calcium oscillations from the Borghans model to stimulate the activation of glycogen phosphorylase. Hence, this is a simple extension of the Borghans model. The model was succesfully tested on MathSBML and Jarnac.</p>\r\n            <br />\r\n            <p>To the extent possible under law, all copyright and related or neighbouring rights to this encoded model have been dedicated to the public domain worldwide. Please refer to      <a href=\"http://creativecommons.org/publicdomain/zero/1.0/\" title=\"Creative Commons CC0\">CC0 Public Domain Dedication</a>\r\n          for more information.      </p>\r\n            <p>In summary, you are entitled to use this encoded model in absolutely any manner you deem suitable, verbatim, or with modification, alone or embedded it in a larger context, redistribute it, commercially or not, in a restricted way or not.</p>\r\n            <br />\r\n            <p>To cite BioModels Database, please use:      <a href=\"http://www.ncbi.nlm.nih.gov/pubmed/20587024\" class=\"external\">Li C, Donizelli M, Rodriguez N, Dharuri H, Endler L, Chelliah V, Li L, He E, Henry A, Stefan MI, Snoep JL, Hucka M, Le Novère N, Laibe C (2010) BioModels Database: An enhanced, curated and annotated resource for published quantitative kinetic models. BMC Syst Biol., 4:92.</a>\r\n                </p>\r\n            </body>\r\n      \r\n    </notes>",
    "files": {
        "additional": [
            {
                "description": "Auto-generated VCML file",
                "fileSize": "36981",
                "name": "BIOMD0000000100.vcml"
            },
            {
                "description": "Auto-generated BioPAX (Level 3)",
                "fileSize": "25621",
                "name": "BIOMD0000000100-biopax3.owl"
            },
            {
                "description": "Auto-generated SBML file with URNs",
                "fileSize": "30392",
                "name": "BIOMD0000000100_urn.xml"
            },
            {
                "description": "Auto-generated Reaction graph (SVG)",
                "fileSize": "19604",
                "name": "BIOMD0000000100.svg"
            },
            {
                "description": "Auto-generated PDF file",
                "fileSize": "179690",
                "name": "BIOMD0000000100.pdf"
            },
            {
                "description": "Auto-generated Scilab file",
                "fileSize": "171",
                "name": "BIOMD0000000100.sci"
            },
            {
                "description": "Auto-generated BioPAX (Level 2)",
                "fileSize": "19165",
                "name": "BIOMD0000000100-biopax2.owl"
            },
            {
                "description": "Auto-generated XPP file",
                "fileSize": "4435",
                "name": "BIOMD0000000100.xpp"
            },
            {
                "description": "Auto-generated Octave file",
                "fileSize": "6855",
                "name": "BIOMD0000000100.m"
            },
            {
                "description": "Auto-generated Reaction graph (PNG)",
                "fileSize": "70380",
                "name": "BIOMD0000000100.png"
            }
        ],
        "main": [
            {
                "fileSize": "31023",
                "name": "BIOMD0000000100_url.xml"
            }
        ]
    },
    "firstPublished": 1174559312000,
    "format": {
        "name": "SBML",
        "version": "L2V1"
    },
    "history": {
        "revisions": [
            {
                "comment": "Original import of Rozi2003_GlycogenPhosphorylase_Activation",
                "submitted": 1174604088000,
                "submitter": "Harish Dharuri",
                "version": 1
            },
            {
                "comment": "Current version of Rozi2003_GlycogenPhosphorylase_Activation",
                "submitted": 1400953919000,
                "submitter": "Harish Dharuri",
                "version": 2
            }
        ]
    },
    "name": "Rozi2003_GlycogenPhosphorylase_Activation",
    "publication": {
        "affiliation": "Department of Physics, Central China Normal University, Wuhan 430079, Hubei, PR China.",
        "authors": [
            {
                "name": "Rozi A"
            },
            {
                "name": "Jia Y"
            }
        ],
        "journal": "Biophysical chemistry",
        "link": "http://identifiers.org/pubmed/14556891",
        "month": "12",
        "pages": "193-202",
        "synopsis": "Taking into account the Ca(2+)-stimulated degradation of inositol 1,4,5-trisphosphate (IP(3)) by a 3-kinase, we have theoretically explored the effects of both simple and complex Ca(2+) oscillations on the regulation of a phosphorylation-dephosphorylation cycle process involved in glycogen degradation by glycogen phosphorylase a-form, respectively. For the case of simple Ca(2+) oscillations, the roles of cytosolic Ca(2+) oscillations in the regulation of active phosphorylase depend upon the maximum rate of IP(3) degradation by the 3-kinase, V(M5). In particular, the smaller the values of V(M5) are, the lower the effective Ca(2+) threshold for the activation of glycogen phosphorylase will be. For the case of complex Ca(2+) oscillations, the average level of fraction of active phosphorylase is nearly independent from the level of stimulation increasing in the bursting oscillatory domain. Both simple and complex Ca(2+) oscillations can contribute to increase the efficiency and specificity of cellular signalling, and some theoretical results of activation of glycogen phosphorylase regulated by Ca(2+) oscillations are close to the experimental results for gene expression in lymphocytes.",
        "title": "A theoretical study of effects of cytosolic Ca2+ oscillations on activation of glycogen phosphorylase.",
        "volume": "106",
        "year": 2003
    },
    "publicationId": "BIOMD0000000100",
    "submissionId": "MODEL4589754842"
}

Using other tools

You have seen a lot of helpful tools for prettifying JSON documents, haven’t you? We just want to summarise that there are an awful lot of other tools like Unix-based commands, Python modules, Node packages or Ruby packages. Let’s discuss all of them in the comment box below this post.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.