Nowadays, an awful lot of web based applications are using JSON as a principal data to load and render information rapidly. While some applications store data into persistence volume as JSON string, the others still use relational databases as the primary storages. Loading data from these relational databases from backend, then building JSON objects and sending them to frontend are all things a web developer often does on a daily basis.
In this post, I will show you how to build JSON objects and pass them to client, such as JavaScript where you have to use the data for displaying or reprocessing.
To ease reading and absorbing the knowledge I am writing here, let’s play with a simple project. The project aims to retrieve organisms and the number of models classified in each organism from BioModels, then make a a bubble chart using D3js library. Thus, you can conceive the idea from getting data from the backend and plotting data to the frontend.
Obviously, the query to fetch all organisms will give you a result in JSON or XML format, so then you can use that format data directly into the view (i.e. frontend) without converting it again. In reality, we often re-process data returned external API before embedding them to internal services. Also, it is very often to obtain data from multiple calls and combine the results before using them in a different place. Moreover, I create a class named Organism to capture the properties such as name, label, taxonomy accession and the number of models for the specific purpose in this post.
Note: This project was built in grails 2.5.6, therefore, it would be valuable if you have experienced and familiarised with it before making your hands dirty.
Table of Contents
Create Organism class
Now, we need to fetch from EBI Search Server data related to BioModels repository. To retrieve data via RESTful API with Grails, I have recently posted an article how to consume web services in Grails which you are free to read and test it. Below is the snippet in BiomodelService to do what has been addressed.
package com.itersdesktop.javatechs.grails
import grails.plugins.rest.client.RestBuilder
import grails.transaction.Transactional
@Transactional
class BiomodelsService {
def fetchAllOrganisms() {
final String BM_SEARCH_URL = "https://wwwdev.ebi.ac.uk/ebisearch/ws/rest/biomodels/topterms"
String queryURL = "${BM_SEARCH_URL}/TAXONOMY?facetfield:label&size=500&format=json"
println queryURL
RestBuilder rest = new RestBuilder(connectTimeout: 10000, readTimeout: 100000, proxy: null)
def response = rest.get(queryURL) {
accept("application/json")
accept("application/xml")
contentType("application/json;charset=UTF-8")
}
if (response.status == 200) {
return response.json
}
return null
}
}
The result we hope to get if the application is running looks like the output of the curl command below.
curl -X GET 'https://wwwdev.ebi.ac.uk/ebisearch/ws/rest/biomodels/topterms/TAXONOMY?facetfield:label&size=500&format=json' -H "application/json"
We expect to receive the output like
{"totalTermCount":1745,"topTerms":[{"text":"9606","docFreq":713},{"text":"4932","docFreq":137},{"text":"10090","docFreq":114},{"text":"40674","docFreq":95},{"text":"131567","docFreq":77},{"text":"562","docFreq":45},{"text":"10114","docFreq":36},{"text":"2759","docFreq":30},{"text":"8355","docFreq":27},{"text":"10116","docFreq":24},{"text":"3702","docFreq":22},{"text":"9986","docFreq":17},{"text":"33090","docFreq":14},{"text":"3701","docFreq":12},{"text":"10141","docFreq":12},{"text":"9615","docFreq":11},{"text":"7711","docFreq":11},{"text":"7227","docFreq":11},{"text":"5691","docFreq":10},{"text":"39107","docFreq":10},{"text":"5141","docFreq":7},{"text":"4896","docFreq":7},{"text":"9913","docFreq":5},{"text":"83333","docFreq":5},{"text":"33208","docFreq":5},{"text":"2242","docFreq":5},{"text":"1773","docFreq":5},{"text":"10088","docFreq":5},{"text":"8782","docFreq":4},{"text":"8292","docFreq":4},{"text":"7742","docFreq":4},{"text":"7215","docFreq":4},{"text":"5658","docFreq":4},{"text":"559292","docFreq":4},{"text":"33154","docFreq":4},{"text":"3193","docFreq":4},{"text":"2208","docFreq":4},{"text":"210","docFreq":4},{"text":"1423","docFreq":4},{"text":"1358","docFreq":4},{"text":"9685","docFreq":3},{"text":"5833","docFreq":3},{"text":"4952","docFreq":3},{"text":"4922","docFreq":3},{"text":"2","docFreq":3},{"text":"11676","docFreq":3},{"text":"11320","docFreq":3},{"text":"10117","docFreq":3},{"text":"10095","docFreq":3},{"text":"9984","docFreq":2},{"text":"9940","docFreq":2},{"text":"9796","docFreq":2},{"text":"9544","docFreq":2},{"text":"7787","docFreq":2},{"text":"746128","docFreq":2},{"text":"70448","docFreq":2},{"text":"6678","docFreq":2},{"text":"6618","docFreq":2},{"text":"64495","docFreq":2},{"text":"6035","docFreq":2},{"text":"5518","docFreq":2},{"text":"5507","docFreq":2},{"text":"5501","docFreq":2},{"text":"5482","docFreq":2},{"text":"5478","docFreq":2},{"text":"5476","docFreq":2},{"text":"5346","docFreq":2},{"text":"5306","docFreq":2},{"text":"5297","docFreq":2},{"text":"5270","docFreq":2},{"text":"5207","docFreq":2},{"text":"520","docFreq":2},{"text":"5180","docFreq":2},{"text":"51453","docFreq":2},{"text":"5062","docFreq":2},{"text":"5061","docFreq":2},{"text":"5057","docFreq":2},{"text":"5037","docFreq":2},{"text":"4959","docFreq":2},{"text":"4924","docFreq":2},{"text":"4897","docFreq":2},{"text":"4837","docFreq":2},{"text":"40563","docFreq":2},{"text":"40559","docFreq":2},{"text":"38033","docFreq":2},{"text":"3704","docFreq":2},{"text":"36914","docFreq":2},{"text":"36911","docFreq":2},{"text":"36630","docFreq":2},{"text":"34099","docFreq":2},{"text":"336810","docFreq":2},{"text":"33188","docFreq":2},{"text":"33178","docFreq":2},{"text":"329376","docFreq":2},{"text":"3055","docFreq":2},{"text":"29883","docFreq":2},{"text":"28985","docFreq":2},{"text":"262014","docFreq":2},{"text":"197043","docFreq":2},{"text":"186490","docFreq":2},{"text":"155892","docFreq":2},{"text":"1496","docFreq":2},{"text":"148305","docFreq":2},{"text":"140110","docFreq":2},{"text":"13684","docFreq":2},{"text":"117187","docFreq":2},{"text":"109871","docFreq":2},{"text":"1063","docFreq":2},{"text":"1047171","docFreq":2},{"text":"104341","docFreq":2},{"text":"9989","docFreq":1},{"text":"9823","docFreq":1},{"text":"9598","docFreq":1},{"text":"9541","docFreq":1},{"text":"9031","docFreq":1},{"text":"85569","docFreq":1},{"text":"83332","docFreq":1},{"text":"7955","docFreq":1},{"text":"7668","docFreq":1},{"text":"7108","docFreq":1},{"text":"70699","docFreq":1},{"text":"7029","docFreq":1},{"text":"672","docFreq":1},{"text":"6499","docFreq":1},{"text":"644223","docFreq":1},{"text":"63577","docFreq":1},{"text":"629395","docFreq":1},{"text":"61434","docFreq":1},{"text":"58853","docFreq":1},{"text":"5850","docFreq":1},{"text":"5791","docFreq":1},{"text":"5782","docFreq":1},{"text":"5548","docFreq":1},{"text":"5544","docFreq":1},{"text":"53533","docFreq":1},{"text":"511145","docFreq":1},{"text":"5076","docFreq":1},{"text":"49990","docFreq":1},{"text":"4930","docFreq":1},{"text":"4929","docFreq":1},{"text":"4894","docFreq":1},{"text":"4892","docFreq":1},{"text":"4547","docFreq":1},{"text":"452646","docFreq":1},{"text":"44689","docFreq":1},{"text":"416870","docFreq":1},{"text":"4113","docFreq":1},{"text":"4097","docFreq":1},{"text":"402676","docFreq":1},{"text":"39782","docFreq":1},{"text":"3847","docFreq":1},{"text":"381512","docFreq":1},{"text":"36420","docFreq":1},{"text":"33169","docFreq":1},{"text":"32524","docFreq":1},{"text":"317","docFreq":1},{"text":"314146","docFreq":1},{"text":"29875","docFreq":1},{"text":"294746","docFreq":1},{"text":"272634","docFreq":1},{"text":"272563","docFreq":1},{"text":"267377","docFreq":1},{"text":"227321","docFreq":1},{"text":"2257","docFreq":1},{"text":"215813","docFreq":1},{"text":"211044","docFreq":1},{"text":"2104","docFreq":1},{"text":"2099","docFreq":1},{"text":"197911","docFreq":1},{"text":"1758","docFreq":1},{"text":"173629","docFreq":1},{"text":"162425","docFreq":1},{"text":"1590","docFreq":1},{"text":"1472294","docFreq":1},{"text":"132504","docFreq":1},{"text":"11292","docFreq":1},{"text":"11276","docFreq":1},{"text":"11103","docFreq":1},{"text":"1034331","docFreq":1},{"text":"10239","docFreq":1},{"text":"10160","docFreq":1},{"text":"101201","docFreq":1},{"text":"10036","docFreq":1},{"text":"10029","docFreq":1},{"text":"10026","docFreq":1},{"text":"1","docFreq":1}]}
From the JSON output, we am going to build a list of Organism object which are rendered as JSON to be sent to views/frontend. The methods are located in BiomodelService under the name toJsonFromString() and buildOrganisms().
def toJsonFromString(final String jsonString) {
def slurper = new JsonSlurper()
slurper.parseText(jsonString)
}
Map buildOrganisms(final String jsonString) {
def parsedJson = toJsonFromString(jsonString)
def totalTermCount = parsedJson.totalTermCount as long
def jsonOrganisms = parsedJson.topTerms
List topTerms = new ArrayList<Organism>()
for (def entry : jsonOrganisms) {
String taxonomy = entry.text
long count = entry.docFreq
String name = resolveTaxonomyTerm(taxonomy)
topTerms.add(new Organism(name: name,
taxonomy: taxonomy, count: count))
}
return [totalTermCount: totalTermCount, topTerms: topTerms]
}
String resolveTaxonomyTerm(String s) {
String serviceURL = "https://www.ebi.ac.uk/ebisearch/ws/rest/taxonomy/entry"
serviceURL = "${serviceURL}/$s?fields=name"
RestBuilder rest = new RestBuilder(connectTimeout: 10000,
readTimeout: 100000, proxy: null)
def response = rest.get(serviceURL) {
accept("application/xml")
contentType("application/xml;charset=UTF-8")
}
if (response.status == 200) {
return response.xml
}
return null
}
def build() {
def jsonString = '''
{"totalTermCount":1745,"topTerms":[{"text":"9606","docFreq":713},{"text":"4932","docFreq":137},{"text":"10090","docFreq":114}]}
// Remember to fetch the full JSON string
def result = buildOrganisms(jsonString)
def organismsMap = result["topTerms"].collect { entry ->
[name: "${entry.name} (${entry.taxonomy})" as String,
count: entry.count as Integer,
taxonomy: entry.taxonomy as String]
}
organismsMap
}
Let’s inspect these methods and understand mechanism behind the scene. At the end of buildOrganism, a list of Organism objects look like
class com.itersdesktop.javatechs.grails.Organism class com.itersdesktop.javatechs.grails.Organism class com.itersdesktop.javatechs.grails.Organism class com.itersdesktop.javatechs.grails.Organism class com.itersdesktop.javatechs.grails.Organism class com.itersdesktop.javatechs.grails.Organism
The second important build is the method build where the list of Organism objects will be converted to a map of looked-like-Organism objects including three properties needed to be populated for the chart.
The list of Organism will be populated to the view, i.e. the bubble nut chart presented in the further section. The key thing here is to build POJOs and render them as JSON in JavaScript, i.e. view/client, where the data will be embedded into charts as well as visualisations. Below is the simple snippet of the output in JSON format.
{
"topTerms": [
{
"name": "Homo sapiens (9606)",
"count": "713"
},
{
"name": "Saccharomyces cerevisiae (4932)",
"count": "137"
},
{
"name": "Mus musculus (10090)",
"count": "114"
},
{
"name": "Mammalia (40674)",
"count": "95"
},
{
"name": "cellular organisms (131567)",
"count": "77"
},
{
"name": "Escherichia coli (562)",
"count": "45"
}
]
}
Now, it is the time to render this data in graphical presentation with d3js. Because of making a bubble chart, the topTerms key will be replaced with children. What you can see in the controller and view looks like
def visualise() {
def organisms = ["children": biomodelsService.build()] as JSON
render(view: "visualise", model: [organisms: organisms])
}
The highlights below should be taken more consideration
def organismsMap = result[“topTerms”].collect { entry ->
[name: “${entry.name} (${entry.taxonomy})” as String,
count: entry.count as Integer,
taxonomy: entry.taxonomy as String]
}
And
def organisms = [“children”: biomodelsService.build()] as JSON
<%@ page contentType="text/html;charset=UTF-8" %>
<html>
<head>
<meta name="layout" content="main"/>
<title>Visualisation of Organisms from BioModels</title>
<script type="text/javascript" src="https://d3js.org/d3.v4.min.js"></script>
</head>
<body>
<div id="organismsChart" class="div-center-content"></div>
<g:javascript>
var dataset = JSON.parse("${organisms}");
console.log(dataset);
var width = 600, height = 600, diameter = 600;
var color = d3.scaleOrdinal(d3.schemeCategory20);
var bubble = d3.pack(dataset).size([diameter, diameter]).padding(1.5);
var svg = d3.select("#organismsChart")
.append("svg")
.attr("width", width)
.attr("height", height)
.attr("class", "bubble");
var nodes = d3.hierarchy(dataset)
.sum(function(d) {
return d.count; });
var node = svg.selectAll(".node")
.data(bubble(nodes).descendants())
.enter()
.filter(function(d){
return !d.children
})
.append("g")
.attr("class", "node")
.attr("transform", function(d) {
console.log(d);
return "translate(" + d.x + "," + d.y + ")";
});
node.append("title")
.text(function(d) {
return d.data.name + ": " + d.data.count;
});
node.append("circle")
.attr("r", function(d) {
return d.r;
})
.style("fill", function(d,i) {
return color(i);
});
node.append("text")
.attr("dy", ".2em")
.style("text-anchor", "middle")
.text(function(d) {return d.data.name.substring(0, d.r / 3);})
.attr("font-family", "sans-serif")
.attr("font-size", function(d) {return d.r/5;})
.attr("fill", "white");
node.append("text")
.attr("dy", "1.3em")
.style("text-anchor", "middle")
.text(function(d) { return d.data.count;})
.attr("font-family", "Gill Sans", "Gill Sans MT")
.attr("font-size", function(d) { return d.r/5;})
.attr("fill", "white");
d3.select(self.frameElement)
.style("height", diameter + "px");
node.on('click', function(d) {
console.log(d);
var label = d["data"]["name"];
var value = d["data"]["count"];
var taxonomy = d["data"]["taxonomy"];
var serverURL = "https://www.ebi.ac.uk/biomodels";
var fixedSearchURL = "/search?domain=biomodels&query=*%3A*+AND+TAXONOMY%3A";
var prefixSearchURL = serverURL + fixedSearchURL;
var queryURL = prefixSearchURL + taxonomy;
window.open(queryURL, '_blank');
});
</g:javascript>
</body>
</html>
Run ./grailsw to start grails wrapper for downloading grails 2.5.6 framework, then start the application by running run-app from grails prompt. Remember to change your application http port if there are multiple grails applications running on your system.
Output
If the application starts properly, accessing the link http://localhost:8181/build-render-json-demo/biomodels/visualise will take you to the page where the bubble chart is shown like the screenshot below.
Source Code
The runnable source code of this article has been pushed to bitbucket as the common place for most illustrated projects for our blogs.
Hopefully, this post would bring to you a convenient approach to build and render JSON elements for views in your Grails applications.
All constructive comments are welcomed. If you are willing to contribute financial support for our website, please follow the instructions below.