Experiments: Data Visualisation in After Effects

Data visualisation over a 3D map of Europe.

Data visualisation fascinates me. I’m amazed by what it can reveal, and the impacts it can have – both positive and negative.

In fact, before I started the masters that I dropped out of this year, I’d originally been looking at an MA in in Design Informatics at Edinburgh – until they increased the price to £12,000. So since I was already dipping into writing scripts for AE, using AE to play with data visualisation was the inevitable next step.

I was recently asked by a client for any ideas on what I could produce for a project with an online gaming company. This seemed like the perfect opportunity to experiment with this idea, so I started learning.

This tutorial by Daniel from Bring Your Own Laptop was all I needed to get started, but he also offers a full course which looks pretty good too!

I decided to put this demo together to share what I’ve learned. You can download the working files here.

How After Effects works with data

Learning to write expressions has definitely helped me gain a new perspective of how After Effects works. When you look at the After Effects Object Model (and then start writing code for it) it becomes very apparent that every element of After Effects is just an object with parameters that can be measured in numbers, arrays of numbers, booleans, and – in the case of text layers – strings. With that in mind, it’s easy to see how any of these parameters can be defined by imported data of the same format.

Getting data into After Effects

Before we can get started, we’ll need to choose some data to work with. For this demo I decided to use some general data about countries of Europe; their population and density, capitals, when/if they joined the EU, that sort of thing – all of which I just pulled from Wikipedia and dropped into a Google Sheets spreadsheet.

AE seems to work best with .json data files. I’m pretty sure it can work with .csv files too but I haven’t tried this. Instead I just copied the data from Google Sheets and pasted into this handy web tool to save a .json file, which I could then import into AE.

Virtually any property of any layer can be linked to a point of data using simple expressions. All we have to do is tell AE which point of data we’re referring to and turn that into a variable we can access:


1
2
3
eval("var *VARIABLE NAME*=" + *DATA FILE NAME*.sourceText);

*NEW VARIABLE NAME* = *VARIABLE NAME*[*DATA INDEX*].*DATA VALUE NAME*;

The parts between asterixis (*) are the parts you’d need to change to suit your own work but here’s what they refer to:

  • Variable name – a generic variable that will hold the whole data file. We use the eval function to convert the file into a usable format (I think? I haven’t actually worked this out completely but it works and seems necessary)
  • Data file name – simply the .json file. I just use the pickwhip to grab this from the project window
  • New variable name – this will become the actual point of data, so a single number, string or boolean
  • Data index – Which data entry you’re referring to (the row in your spreadsheet)
  • Data value name – which point of data from that entry we’re referring to (the column in your spreadsheet)

So in my own example, if I’m looking to get the population density data for the first country in my list, that expression would look something like this:


1
2
eval("var pop=" + footage("europe2.json").sourceText);
dens = pop[0].Density;

Now I have a variable called “dens” that I can use to define the value for my property.

Other features that will help

So now we have our data, what do we do with it? The easiest step is using this data to set the position, scale or rotation of our layer. For example, if I just add one more like to that expression:


1
dens;

that property value would now be the same as the data. If the property requires more than one value (eg position requires an x and a y value), we could write it as an array:


1
[dens, value];

This would set the x position to whatever the data is, and the y position to whatever value it was already set to.

Nice and simple, but lets go a little further.

Animation

Simply setting the value to be the same as the data won’t make for exciting animation. We need something to control the value so that we can set keyframes, so a simple slider (Effects & presets > Expression controls > Slider Control) can do this. All we have to do is add this as an effect to our layer, then set some keyframes changing it’s value from, say, 0 to 1, then multiply our variable by that:


1
dens = pop[0].Density * effect("Slider Control")("Slider");

Now the “dens” value will go from 0 to whatever it’s value is multiplied by 1 as the slider moves between keyframes.

“Points follow nulls”

The 2018 update to After Effects CC introduced a very useful tool that will definitely help with animating data. “Create nulls from paths” can be found at the bottom of the “Window” toolbar list if it’s not already open. Select the path you want to use then hit “points follow nulls” – AE will create a null object for each anchor point on the path, meaning you can manipulate the position of each point individually; great for line graphs or bar charts.

Working with colour

Colour can be a very useful tool when visualising data (eg heat maps) but this can be a little more complicated in After Effects since colours are defined in a slightly different way. You could theoretically use the data to define the red/green/blue values of your colour, or the opacity, but if you want to create a nice gradient there’s another way – the “sampleImage” function. This basically works as a colour picker on a specified layer (which can be in a different composition). All we have to do is say which layer we’re targeting, and the x and y coordinates. Create a simple linear gradient as a layer and set that as the target, then use the data to define the x or y coordinate (depending on which way the gradient runs) and you’ve got yourself a data defined colour! As an example, I created a simple gradient from white to pink – top to bottom, then used the population density data from before to pick a point on that gradient:


1
2
3
4
5
6
7
8
9
10
11
//bring in the data file
eval("var pop=" + footage("europe2.json").sourceText);

//define the target for sampling
target = comp("gradient").layer("gradient.ai");

//create a variable from the data
grad = pop[0].Density * effect("Slider Control")("Slider");

//use the variable to define the location of the sample
value = target.sampleImage([5, grad]);
Visualisation of how the sampleImage function works. Belgium's density is 374.2 so we pick the colour on the target at coordinates 5, 374.2. The result is the fill turns pink to match the colour on the scale.

Things to keep in mind

Good data viz vs bad data viz

Data visualisation is a skill within itself; it takes a good understanding of visual communication to design it well and it’s very easy to get it wrong – there’s a lot of bad examples out there. Data visualisation is all about turning flat data into something visual, to help us recognise patterns and information that might otherwise be invisible when looking at a list of numbers. As with all design, it has a function; it’s not just about making something pretty. It should give insight, help communicate a message, and most importantly should be easy to understand. I’d recommend having a look through some examples (not necessarily animated) with a critical eye. There’s lots of examples on Information is Beautiful (not all necessarily good), and Reddit has some great discussions on /r/dataisbeautiful and /r/dataisugly.

Data viz in After Effects can be a slog

There are many different ways to represent the same data, so it’s important to find the right one. This can often mean a lot of trial and error but this comes up against a simple issue; working with data can make AE crawl. For each frame, After Effects is opening the data file, finding the point of data, putting that into whatever expression you’ve created and then setting the value of that property, and it’s doing this for each object that you’re animating with data before turning all of that into a matrix of pixels. Working with countries of Europe meant I was working with 41 objects; that’s a lot of data for AE to churn through 25 times a second.

I’d recommend making some quick graphs or charts in Excel or Google Sheets and drawing up your ideas before you start creating them in AE. The last thing you want is to create all of that animation only to find it looks ugly or unreadable.

Exporting your animation

From what I’ve worked out, Adobe Media Encoder will not accept data when exporting. The easiest way around this I found was to export as a .avi or .mov through the render queue into a folder which is set as a watch folder for AME. That way you can still have create a .mp4 file if you need it.

Examples

Population density heat map

The first example I produced was a heat map showing the population density of each country. To start, I’d need a basic map of Europe with each country as a separate layer which I could then import into AE as shape layers.

Drawing around fjords is a nightmare. Norway I’m looking at you…

With each country as a shape layer, I could now define the fill colour using the expression from before:


1
2
3
4
5
6
7
8
9
10
11
//bring in the data file
eval("var pop=" + footage("europe2.json").sourceText);

//define the target for sampling
target = comp("gradient").layer("gradient.ai");

//create a variable from the data
grad = pop[0].Density * effect("Slider Control")("Slider");

//use the variable to define the location of the sample
value = target.sampleImage([5, grad]);

Then I’d just have to repeat this for each layer/country. One potential issue I could have (and did) run into here was making sure the data matched up to the layer. I did this by putting the data and the layers in alphabetical order by country name. Then all I had to do was copy and paste the expression into each layer, changing the index number (pop[1].Density, pop[2].Density etc) for each layer. Repeating this 41 times was tedious to say the least, and made it easy to make errors. Another way around this would have been to use a simple for loop that would compare the layer name with that data index “country name” data value, and if it matched, use that index’s density data. This would have made the process quicker and reduced the risk of errors, but in turn would have massively reduced rendering times, so it’s a double edged sword.

I also wanted to create a simple bar graph to accompany the heat map, showing the actual values of each country. Something like this:

Graphs of population density

To do this, I simply created lines on shape layers, then used the “points follow null” script to define the x position of the second point, relative to the first, thus defining the length of the line


1
2
3
4
5
6
7
8
9
10
11
12
13
14
//bring in the data
eval ("content =" + footage("europe2.json").sourceText);

//define the value to be used as a variable
pop = content[0].Density;

//x position is the same as the x position of the first null, plus the value of the "pop" variable (multiplied by a slider for animation)
x = thisComp.layer("Shape Layer 1: Path 1 [1.1.0]").transform.position[0] + (pop * effect("Slider Control")("Slider"));

//y is the same as the other null - keeping the line horizontal
y = thisComp.layer("Shape Layer 1: Path 1 [1.1.0]").transform.position[1];

//set the value for the property
[x,y];

The text simply took the country name and the population density number and combined them together into a string. The majority of this expression is actually just there to put commas in numbers over 1,000 (sourced from here).


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
//bring in the data
eval ("content =" + footage("europe2.json").sourceText);

//define the country and density values
country = content[0].Country;
pop = content[0].Density * effect("Slider Control")("Slider");

//this removes excessive decimal places that appear due to the slider moving between values
pop = pop.toFixed(2);

//add commas...
numAsText= ""+pop;
dotIndex = numAsText.indexOf("."); // check for decimals
if (dotIndex==-1) { dotIndex=numAsText.length; }

formattedText = numAsText.substring(dotIndex-3,dotIndex+3); // +4 for 3 decimals etc.

if (dotIndex>3) {
    for (i=dotIndex-3 ; i>0 ; i=i-3) {
        extractedText = numAsText.substring(i-3,i);
        if (extractedText=="-") {
            formattedText = extractedText + formattedText
        } else {
            formattedText = extractedText + "," + formattedText
        }
    }
}

//create string
country + ": " + formattedText;

Put those together and you get something a bit like this:

Not necessarily the best animation you’ve ever seen, but a proof of concept at least.

3D population maps

Next I wanted to see whether I could take this into 3D. I took that heat map composition and added an extrusion (found in geometry settings if you turn the shapes into 3D layers, and you’re using the ray-traced or Cinema 4D renderer), with the extrusion value set by the population. This way we could compare density with population. The expression goes something like this:


1
2
3
4
5
6
7
8
//bring in the data
eval ("content =" + footage("europe2.json").sourceText);

//set the variable - I divided by 250,000 so that we didn't end up with values in the millions...
pop = (content[0].Population/250000) * effect("Slider Control")("Slider");

//set the value to be the variable
pop;

Repeat for each country (and add the same expression to the z position), add a camera and some lights and you get this…

Oh dear…

This took an eternity to render since each shape is so complicated, and ultimately it just made everything difficult to see. Lesson learned, I decided instead to show population density in capital cities instead. This meant simply creating a new illustrator file with circles to represent the cities, then applying the same expression to those circles. I then attached a text layer to each column, to see numbers and names. Originally I wanted this text to be included as 3D layers that would then orient themselves to face the camera, but with rotation values changing so much this didn’t work so good. Instead I had to leave the text in 2D. This would normally cause a problem since 2D layers can’t be parented to 3D ones, however a simple expression meant they would always match position:


1
thisComp.layer("Albania Outlines 2").toComp([0, 0, 0]);

The result was better.

Still, quite unreadable – especially the text – and I’m not really sure how easy it is to compare the colour of the country with the height of the column, but it was an interesting experiment.

Animating time dependent data

Finally, I wanted to animate time based data. This gets a little more tricky since expressions are read on every frame, and you can’t keyframe them. There are ways to link expressions to set keyframes (“at this keyframe, do this”) but since I want the data to set the timing for me, I can’t create the keyframes myself. The solution – more of a workaround really – was to use the frame number as another point of data for comparison.

For my example, I wanted to show when countries joined the EU, having their shape fill with colour on the year specified. I needed a counter that would show the current year, and then each shape would need an expression saying “if this is the same, show the fill”. For the counter, I used the numbers effect on a text layer, set the value to 1954 (the lowest value) then used an expression to add the current frame number to this, so that it increased by 1 on each frame:


1
value + timeToFrames(time);

The fill basically just took the year from the data, then compared this with the current value of that text layer. If the value was equal to or higher than the data value, opacity was set to 100%.


1
2
3
4
5
6
7
8
9
10
11
//get the data
eval ("content =" + footage("europe2.json").sourceText);

//set a variable for the year
year = content[2].Joined;

//compare this with the value of the numbers effect, if it's the same or higher, set opacity to 100, otherwise leave it at 0.
if (thisComp.layer("").effect("Numbers")("Value/Offset/Random Max") >= year)
  { 100; }
else
  { 0; }

Put those together and you end up with something like this:

via Gfycat

I had to reduce the framerate to make sure anything was actually visible. The alternative was using an if statement that said something like “if the frame number divided by 10 is the same” but this brought up all sorts of rounding and decimals and whatever other complications that I didn’t want to deal with for the sake of a demo.

Adding the country names was simply a process of just setting their entry point manually, but this could have been done with expressions too if I could have been bothered. Similarly, the animation could be much more exciting, but again since I can’t use expressions to say “put this keyframe at this time” they too would have to do something like “increase opacity by 10% on each frame”, which then wouldn’t work well because of the reduced framerate. You can see problems start emerging quickly.

Still, this was a worthy experiment and I’m quite happy with what I’ve learned from it. But there’s lots more to do.

Conclusions

So I’ve learned from this that it’s quite easy to bring data into After Effects. One great feature that I haven’t demonstrated here, is that it’s also very easy to swap that data later on. I could just replace the .json file and all of the expressions would automatically update. So long as the same data sets can be found, the animation will update. This is useful if you’re maybe making something each week, like results from a sports/gaming league, or news headlines etc.

The process can be slow, even tedious at times, and getting something that looks good takes a lot of skill (I wouldn’t necessarily say that any of these results look good – they were merely for the sake of learning the techniques), but AE is a powerful tool, and it’s even more powerful with data.

Next steps

Moving on from here, I’m thinking about taking this into scripts. Can scripts pick up data from .json files? Could I use scripts to set keyframes in a way that isn’t possible with expressions? Could scripts speed up the laborious process of copying and pasting the same expression on 41 different layers?

There’s lots of potential here. What are your thoughts?

Share
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.