Dashboards in Your Browser: Lessons From Working With d3.js
d3.js is a pain in the backside to learn. I’m here to share some helpful lessons that I learned while trying to pick it up and hopefully save you a few precious hours of frustration.
You can see and try out these tips for yourself — the code snippets provided are taken from this Github repository. To run the code: navigate to the project folder in your terminal and run python app.py (requires python to be installed). Your terminal will run the visuals on a localHost server.
If you’re completely new to d3, I’d recommend checking out a couple of beginners tutorials before getting stuck in to a complete visual. Try this article for a nice overview.
When learning a new data tool, I like to work with football data, since I know and understand it reasonably well. This example uses data from the 2017–2018 Premier League season provided by Football Reference.
Lesson 1: Pass only the data that you require to the web page.
This example uses Flask as a back-end. Flask is a micro web framework written in python. I won’t go into detail about Flask in this post; in this case it renders, and passes data to, the web page.
Why am I using Flask?
Your data takes up memory. The more memory your webpage requires, the longer it will take to render and to function. And yes, I learnt this by trying to dump my entire data file into my d3 viz and waiting forever to see the goods.
Using Flask, I’m able to utilize forms in my web page to request only the parts of the dataset that I require. In our example, we’re only ever comparing two Premier League football teams at a time. Hence, I only need to pass data for those two teams to the web page:
I value versatility. If you’ve spent more than a few hours angrily trying to learn how to use d3 (it’s the only way), then you will too.
Lesson 2: Buttons are your friend
Using buttons, one chart can become many. You’ll have noticed in the gif that by clicking on the ‘Goals’ or ‘Assists’ buttons, the variables presented on the scatter plot are changed and the axes updated. Here’s how they work:
d3.select(“#goalscoringBtn”) // Find the button in our HTML code
.on(“click”, function() { // When button is clicked, run function
xVal = ‘Goals’ // Change axis value variables
yVal = ‘xG’
updateChart(xVal,yVal) //Call update function with new values
});d3.select(“#creatingBtn”)
.on(“click”, function() {
xVal = ‘Ast’
yVal = ‘xA’
updateChart(xVal,yVal)
});
Simple!
Lesson 3 (a lesson in life): Don’t try to reinvent the wheel
I had an idea. I wanted to be able to see data from different periods in the season. Standard date range filters bore me. More importantly, I can’t see what was going on over the period I select if all I have is a date range.
I’d seen some examples of people using d3.brush to zoom in visuals. I liked the idea of being able to see the results over the course of the season and just being able to click and drag over those I wished to include. You can see where this is going: Brush filters! Here is how the brush filter works:
var extent = [0, 10000] // This sets the default range to include
// all dates in our datasetsvgForm.append(“g”).attr(“class”, “brush”)
.call( d3.brushX() // Add the brush feature using the d3.brush
// function on the x axis only
.extent([[0,0], [formWidth,formHeight]]) // initialise brush area
.on(“end”, updateAll) // Each time the brush selection changes,
// trigger the ‘updateChart’ function
).lower(); // pushes the brush down a level (behind the dots)function updateAll() {
updateChart(xVal,yVal); // Call the update function with current
// axis vars
updateChart(xVal,yVal);
}
You have a question. Why am I calling updateChart twice?
Frankly, it’s a bit of a workaround. When I use the brush filter, updateChart is called and all proceeds as one might expect. Except if there are dots to be added. The second call appends these fresh dots from the enter selection.
Implementing the brush filter means that we need date-level granularity in our dataset. We want to present this data as an aggregate through our final visual.
Lesson 4: Grouping data — d3.nest
Grouping is actually quite straightforward using d3. You’ll use the following functions regularly. Save yourself a copy so that you can slot them straight into future projects.
Anyway, specify a key to group the data by and rollup any fields you’d like to include:
//nest main dataset by player
var nested = d3.nest()
.key(function(d) {return d.Player})
.rollup(function(player) {
return player.map(function(c) {
return {“Gls”: +c.Gls,’Min’:+c.Min, “xG”: +c.xG,’Ast’:
+c.Ast,’xA’:+c.xA, ‘DateParse’: c.DateParse, “Squad”:
c.Squad,”P_color”: c.P_color, “S_color”:c.S_color
}
});
}).entries(records) //input datasetNext, you’ll need to sum the data in each group. Another handy function:
//Sum up stats in the nested data
function sumStatsScatter(node) {
if (node.value) {
node.values = node.value;
delete node.value;
}
node.Goals = node.values.reduce(function(r,v){
return r + (v.value? sumStats(v) : v.Gls);
},0);
node.xG = node.values.reduce(function(r,v){
return r + (v.value? sumStats(v) : v.xG);
},0);
node.xA = node.values.reduce(function(r,v){
return r + (v.value? sumStats(v) : v.xA);
},0);
node.Ast = node.values.reduce(function(r,v){
return r + (v.value? sumStats(v) : v.Ast);
},0);
node.Min = node.values.reduce(function(r,v){
return r + (v.value? sumStats(v) : v.Min);
},0);
return node.Goals, node.xG,node.Ast, node.xA,node.Min
}
//Call function on nested data
nested.forEach(function(node) {
sumStatsScatter(node)
})There’s something missing here — we’re pressing buttons and applying filters and re-grouping data. Where are we doing that?
Lesson 5: Try to understand, or at least learn to use, the d3 update cycle
The process for updating a d3 viz when applying filters can be confusing at first. It’s worth taking the time to work through some examples and gain some understanding of what’s going on before you sink countless hours of frustration into winging it (à la this guy).
Update functions allow us to filter, transition and even redraw components of our visual. That’s powerful. Let’s go through the update cycle of our example step-by-step:
//Update function
function updateChart(xVal,yVal) {
extent = d3.event.selection //used to find selected date range
if(!extent){
extent = [0, 10000] //if no date range selected, take all
// available
}//Filter the dataset on the range selected by the brush filter:
var dataFilter = records.filter(function(d) {return d.DateParse >=
x.invert(extent[0]) & d.DateParse <= x.invert(extent[1])})
So, our update function has been called. First things first. Get your data in order. Using x.invert(extent…) we can convert our brushed range into a min and max date. Filter your data on these dates and re-group with a fresh call of d3.nest.
Next, get those axes in order: find your new data range, redefine your scales and redraw your labels. Here’s a snippet of the x axis update:
// Find the new max x/y value. I want both axes to have the same
// range, so I find the max of the two axes.
var scatterDomMax = d3.max(nestedFilter,function(d){return (d[xVal]
/ (d.Min/90))})*1.1 > d3.max(nestedFilter,function(d){return
(d[yVal]/(d.Min/90))})*1.1 ? d3.max(nestedFilter,function(d)
{return (d[xVal] / (d.Min/90))})*1.1 : d3.max(nestedFilter
,function(d){return (d[yVal]/(d.Min/90))})*1.1// Redefine define x scale
var xScale = d3.scaleLinear()
.domain([0,scatterDomMax])
.range([margin.left, 400]);// Call your new axis with a transition
var xAxis = svgScatter.selectAll(“g.x-axis”)
.transition().duration(500)
.call(d3.axisBottom(xScale))// Remove old axis label
svgScatter.selectAll(“.scatterLabels”).remove();// Add new axis label
svgScatter.append(“text”)
.attr(“class”,”scatterLabels”)
.style(“text-anchor”, “end”)
.attr(“x”, 405)
.attr(“y”, 395)
.style(“font-size”,”24px”)
.style(“fill”,generalTextColor)
.text(xVal + ‘/90’).lower(); // pushes text back so that we can
// still hover over dots
Finally, let’s talk about the enter-exit-remove update cycle of d3:
// Select all of the circles currently on our scatter plot and
// update the data source
var updatedDots = d3.select(“#scatter-chart”).select(“#svg-”).selectAll(“circle”).data(nestedFilter);// Enter our new data and append any new circles required.
updatedDots.enter().append(“circle”);// Remove any circles which no longer have data bound
updatedDots.exit().remove();// We will need to redraw our tooltip with updated data
var tooltip = d3.select(“#tooltip”).style(“opacity”,0);// Add a transition to move the dots to their new positions, add
// or remove dots as necessary
updatedDots
.transition()
.duration(1000)
.attr(“cx”, function (d) { return xScale(d[xVal]/(d.Min/90)); })
.attr(“cy”, function (d) { return yScale(d[yVal]/(d.Min/90)); })// etc...
There’s quite a lot going on there. Let’s break it down step-by-step, as I understand it:
- Select all of the points on the scatter plot. Add our filtered data. New data points are bound to available dots.
- enter() stores any data points from our updated data which didn’t have an item to bind to. We then append any extra dots that we require such that each data point has its own dot.
- exit() stores any excess dots after data has been assigned, which can be removed from the visual by calling .remove().
- Transition the dots, each bound to its own new data point, to their updated positions.
There you have it. These five little lessons have helped me tremendously in getting interactive visuals to function relatively quickly. I’d recommend cloning the Github repo and having a play around with it yourself.
For an example of a more complete interactive d3 dashboard-style page, feel free to check this out.
