Blog / Animated syntax highlighting, 4k 60 FPS edition

An animated syntax highlighter for the web is nice and all, but how about rendering animated, highlighted code to high-quality video?

This post shows how a relatively simple script for Node.js can do just that! If you don't care about the details just copy and paste this Gist. If you do care, read on...

Preparations

The general approach for our script is as follows:

1. take regular animation data from @codemovie/code-movie and manually pre-compute every frame
2. use the browser automation library Puppeteer to render the animation as per usual, using HTML and CSS
3. cycle through each animation frame and take screenshots every time
4. pump the screenshots into ffmpeg using the image2pipe muxer and encode the final result as a video

The end result will amount to a janky script that abuses just about every part of its pipeline, but it will work! To get started, we need Node.js >= 20 and a somewhat recent version of ffmpeg installed. Once that's done, we need at least the following package.json to make sure that we can use top-level await in our script:

{
  "type": "module"
}

{
  "type": "module"
}

Then we can install the required NPM dependencies:

• @codemovie/code-movie @ ~0.0.16
• puppeteer
• bezier-easing

npm i @codemovie/code-movie puppeteer bezier-easing

npm i @codemovie/code-movie puppeteer bezier-easing

After installation, it's time to go scripting!

The project

For maximum flexibility we should store the actual animation project in a JSON file that the main script can import:

{
  "languageModule": "json",
  "languageOptions": {},
  "frames": [
    {
      "code": "[]"
    },
    {
      "code": "[\"World\"]"
    },
    {
      "code": "[\"Hello\", \"World\"]"
    },
    {
      "code": "[\n  \"Hello\",\n  \"World\"\n]"
    }
  ],
  "videoOptions": {
    "fps": 60,
    "width": 1920,
    "height": 1080,
    "transitionTime": 500,
    "pauseTime": 1000
  }
}

{
  "languageModule": "json",
  "languageOptions": {},
  "frames": [
    {
      "code": "[]"
    },
    {
      "code": "[\"World\"]"
    },
    {
      "code": "[\"Hello\", \"World\"]"
    },
    {
      "code": "[\n  \"Hello\",\n  \"World\"\n]"
    }
  ],
  "videoOptions": {
    "fps": 60,
    "width": 1920,
    "height": 1080,
    "transitionTime": 500,
    "pauseTime": 1000
  }
}

If you have used Code.Movie before, this should broadly make sense:

• "languageModule" and "languageOptions" refers to a language module and its options
• "frames" is the list of code keyframes
• "videoOptions" is, unsurprisingly, a set of options for the video; frame rate, resolution, animation transition time and the amount of milliseconds the animation should pause on each keyframe

This information now just needs to travel through the pipeline of Code.Movie, puppeteer, and ffmpeg.

The main script

The main script first needs to import the project file:

import project from "./project.json" with { type: "json" };
const {
  frames,
  languageModule,
  languageOptions,
  videoOptions,
} = project;

import project from "./project.json" with { type: "json" };
const {
  frames,
  languageModule,
  languageOptions,
  videoOptions,
} = project;

After that, we need an HTML scaffold to contain the animation in puppeteer. This contains a bit of CSS and JavaScript to scale the animation to the available screen size:

const baseHTML = `<style>
  body {
    /* this font-size ensures ok looking text no matter the scale */
    font-size: 48px;
    /* this script pre-computes all animations */
    --cm-animation-duration: 0s;
    margin: 0;
    padding: 0;
  }
  /* Ensures the animation is centered in the frame */
  .cm-animation {
    transform-origin: top left;
    position: absolute;
    top: 50%;
    left: 50%;
  }
</style>
<script>
  const root = document.querySelector(".cm-animation");
  const { offsetWidth, offsetHeight } = root;
  const { innerWidth, innerHeight } = window;
  const scale =
    Math.min(innerWidth / offsetWidth, innerHeight / offsetHeight);
  root.setAttribute(
    "style",
    "transform:scale(" + scale + ") translate(-50%, -50%)"
  );
</script>`;

const baseHTML = `<style>
  body {
    /* this font-size ensures ok looking text no matter the scale */
    font-size: 48px;
    /* this script pre-computes all animations */
    --cm-animation-duration: 0s;
    margin: 0;
    padding: 0;
  }
  /* Ensures the animation is centered in the frame */
  .cm-animation {
    transform-origin: top left;
    position: absolute;
    top: 50%;
    left: 50%;
  }
</style>
<script>
  const root = document.querySelector(".cm-animation");
  const { offsetWidth, offsetHeight } = root;
  const { innerWidth, innerHeight } = window;
  const scale =
    Math.min(innerWidth / offsetWidth, innerHeight / offsetHeight);
  root.setAttribute(
    "style",
    "transform:scale(" + scale + ") translate(-50%, -50%)"
  );
</script>`;

It is important to set --cm-animation-duration to 0s because this script pre-computes all animation information and Code.Movie's usual CSS animations could get in the way of that - so better switch them off.

Next: start puppeteer and ffmpeg!

import { launch } from "puppeteer";
import { spawn } from "node:child_process";

const browser = await launch({
  headless: true,
  args: [
    "--hide-scrollbars",
    "--no-default-browser-check",
    "--no-first-run",
  ],
});

const ffmpeg = spawn("ffmpeg", [
  "-loglevel",
  "error",
  "-r",
  `${videoOptions.fps}`,
  "-f",
  "image2pipe",
  "-s",
  `${videoOptions.width}x${videoOptions.height}`,
  "-i",
  "pipe:0",
  "-vcodec",
  "libx264",
  "-crf",
  "25",
  "-pix_fmt",
  "yuv420p",
  "-f",
  "ismv",
  "pipe:1",
]);

import { launch } from "puppeteer";
import { spawn } from "node:child_process";

const browser = await launch({
  headless: true,
  args: [
    "--hide-scrollbars",
    "--no-default-browser-check",
    "--no-first-run",
  ],
});

const ffmpeg = spawn("ffmpeg", [
  "-loglevel",
  "error",
  "-r",
  `${videoOptions.fps}`,
  "-f",
  "image2pipe",
  "-s",
  `${videoOptions.width}x${videoOptions.height}`,
  "-i",
  "pipe:0",
  "-vcodec",
  "libx264",
  "-crf",
  "25",
  "-pix_fmt",
  "yuv420p",
  "-f",
  "ismv",
  "pipe:1",
]);

The former has a proper JavaScript API while the latter is just a process with I/O streams. To get the ffmpeg process to write a file, we need to connect its stdout to a write stream:

import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { createWriteStream } from "node:fs";
const output = join(
  fileURLToPath(import.meta.resolve("./")),
  "movie.mp4",
);
ffmpeg.stdout.pipe(createWriteStream(output));

import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { createWriteStream } from "node:fs";
const output = join(
  fileURLToPath(import.meta.resolve("./")),
  "movie.mp4",
);
ffmpeg.stdout.pipe(createWriteStream(output));

With the pipeline in place, we can start to use it as planned! First, get Code.Movie imported and set up according to the data from the project file:

import {
  fromStringsToScene,
  toAnimationHTML,
} from "@codemovie/code-movie";
const language = (
  await import(`@codemovie/code-movie/languages/${languageModule}`)
).default(languageOptions);

// Create a scene object with the language object as usual
const { scene } = fromStringsToScene(frames, {
  language,
  tabSize: 2,
});

import {
  fromStringsToScene,
  toAnimationHTML,
} from "@codemovie/code-movie";
const language = (
  await import(`@codemovie/code-movie/languages/${languageModule}`)
).default(languageOptions);

// Create a scene object with the language object as usual
const { scene } = fromStringsToScene(frames, {
  language,
  tabSize: 2,
});

Scene objects usually only contain keyframes and leave interpolation between keyframes to the browser's CSS engine. For this script we will need to pre-compute the animation information for every frame ourselves, so that we can have precise control over the render-screenshot-loop. This pre-computation happens in an extra module that we will come back to shortly. For now, let's simply assume that interpolateFrames() does all the required calculations and simply updates the scene object accordingly:

import { interpolateFrames } from "./ease.js";
interpolateFrames(
  scene,
  videoOptions.fps,
  videoOptions.transitionTime,
  true,
);

import { interpolateFrames } from "./ease.js";
interpolateFrames(
  scene,
  videoOptions.fps,
  videoOptions.transitionTime,
  true,
);

Because the update scene object is still just a scene object, we can shove it into toAnimationHTML() as if we were still working on an animation for the web:

const animationHtml = toAnimationHTML({ scene, language });

const animationHtml = toAnimationHTML({ scene, language });

Thanks to the pre-computed animations the HTML string (containing tens of thousands of CSS rules) is now much larger than usual. But that's fine, today's browsers can almost handle this kind of abuse:

const page = await browser.newPage();
await page.setViewport({
  width: videoOptions.width,
  height: videoOptions.height,
});
const pageContent = "<!doctype html>" + animationHtml + baseHTML;
await page.setContent(pageContent);

const page = await browser.newPage();
await page.setViewport({
  width: videoOptions.width,
  height: videoOptions.height,
});
const pageContent = "<!doctype html>" + animationHtml + baseHTML;
await page.setContent(pageContent);

Don't worry about the bug in Chrome, @codemovie/code-movie >= 0.0.16 contains a workaround for this particular limitation.

It is important to add the HTML scaffold after the animation HTML to make sure that the animation DOM is already parsed when the script inside the scaffold runs its scaling logic. Now we just need to calculate more constants...

// To add pauses on keyframes, simply pump the relevant frame
// repeatedly into ffmpeg.
const extraKeyframeFrames =
   (videoOptions.pauseTime / 1000) * videoOptions.fps;

// Stringify the number of total frames so we can use this string
// length to pad the current frame number for stdout.
const totalFrames = String(scene.frames.size);

// To add pauses on keyframes, simply pump the relevant frame
// repeatedly into ffmpeg.
const extraKeyframeFrames =
   (videoOptions.pauseTime / 1000) * videoOptions.fps;

// Stringify the number of total frames so we can use this string
// length to pad the current frame number for stdout.
const totalFrames = String(scene.frames.size);

... before we can run the main loop:

// For every frame, set the correct frame index via JS, then
// render a PNG buffer and hand it over to ffmpeg's stdin.
// Rinse and repeat.
for (const [frameIdx, { isKeyframe }] of scene.frames) {
  // Switch the frame
  await page.evaluate(
    `document.querySelector(".cm-animation")
      .classList
      .remove("frame${frameIdx - 1}");
document.querySelector(".cm-animation")
  .classList
  .add("frame${frameIdx}");`,
  );
  // Waste some time to allow the browser to update. You may
  // need to adjust this, or this might be entirely superfluous
  // depending on your machine, software and project. 100ms
  // works for me and my machine ¯\_(ツ)_/¯
  await new Promise((r) => setTimeout(r, 100));
  // Take the actual screenshot
  const buffer = await page.screenshot({
    type: "png",
    optimizeForSpeed: true,
  });
  ffmpeg.stdin.write(buffer);
  // Pause on keyframes and the last frame
  if (isKeyframe || frameIdx === scene.frames.size - 1) {
    for (let i = 0; i < extraKeyframeFrames; i++) {
      ffmpeg.stdin.write(buffer);
    }
  }
  const currentFrame =
    String(frameIdx + 1).padStart(totalFrames.length, "0");
  console.log(`Rendered frame ${currentFrame} of ${totalFrames}`);
}

// For every frame, set the correct frame index via JS, then
// render a PNG buffer and hand it over to ffmpeg's stdin.
// Rinse and repeat.
for (const [frameIdx, { isKeyframe }] of scene.frames) {
  // Switch the frame
  await page.evaluate(
    `document.querySelector(".cm-animation")
      .classList
      .remove("frame${frameIdx - 1}");
document.querySelector(".cm-animation")
  .classList
  .add("frame${frameIdx}");`,
  );
  // Waste some time to allow the browser to update. You may
  // need to adjust this, or this might be entirely superfluous
  // depending on your machine, software and project. 100ms
  // works for me and my machine ¯\_(ツ)_/¯
  await new Promise((r) => setTimeout(r, 100));
  // Take the actual screenshot
  const buffer = await page.screenshot({
    type: "png",
    optimizeForSpeed: true,
  });
  ffmpeg.stdin.write(buffer);
  // Pause on keyframes and the last frame
  if (isKeyframe || frameIdx === scene.frames.size - 1) {
    for (let i = 0; i < extraKeyframeFrames; i++) {
      ffmpeg.stdin.write(buffer);
    }
  }
  const currentFrame =
    String(frameIdx + 1).padStart(totalFrames.length, "0");
  console.log(`Rendered frame ${currentFrame} of ${totalFrames}`);
}

Once the loop is over, we close the input stream on the ffmpeg process and have puppeteer terminate the headless browser. Almost done!

ffmpeg.stdin.end();
await browser.close();

ffmpeg.stdin.end();
await browser.close();

The only missing bit is the frame interpolation module ease.js that we previously glossed over.

Frame interpolation

The structure of a scene object is as of Code.Movie 0.0.16 roughly as follows:

type Scene = {
  objects: {
    // ID -> content
    text: Map<number, TokenOutput>;
    // ID -> content
    lineNumbers: Map<number, LineNumberOutput>;
  };
  // Frame number -> Frame
  frames: Map<number, Frame>;
  // Frame number -> Frame metadata
  metadata: Map<number, FrameMetadata>;
};

type Scene = {
  objects: {
    // ID -> content
    text: Map<number, TokenOutput>;
    // ID -> content
    lineNumbers: Map<number, LineNumberOutput>;
  };
  // Frame number -> Frame
  frames: Map<number, Frame>;
  // Frame number -> Frame metadata
  metadata: Map<number, FrameMetadata>;
};

objects contains the content of rendered objects, such as the text and type of text tokens. Every object has an ID by which it is uniquely identified within a scene. frames contains the position information of every object for every scene, with each frame roughly looking like this:

type Frame = {
  isKeyframe: boolean;
  cols: number;
  rows: number;
  text: Map<number, TextPosition>; // ID -> position
  lineNumbers: Map<number, { a: number }>; // Index -> alpha
};

type Frame = {
  isKeyframe: boolean;
  cols: number;
  rows: number;
  text: Map<number, TextPosition>; // ID -> position
  lineNumbers: Map<number, { a: number }>; // Index -> alpha
};

By default, every frame is a keyframe and every text position (an object of type { x: number; y: number; a: number }) and line number alpha value is a whole number - Code.Movie simply relies on the browser's CSS engines to do the actual animating work.

To pre-compute the animation, we first have to setup the easing function that interpolates x, y and/or a for each frame:

// Set up the easing function
import BezierEasing from "bezier-easing";
const ease = BezierEasing(0.25, 0.1, 0.25, 1);

// Interpolate the value between "from" and "to" for a
// given "step" of "steps"
function value(from, to, step, steps) {
  return from + (to - from) * ease(step / steps);
}

// Set up the easing function
import BezierEasing from "bezier-easing";
const ease = BezierEasing(0.25, 0.1, 0.25, 1);

// Interpolate the value between "from" and "to" for a
// given "step" of "steps"
function value(from, to, step, steps) {
  return from + (to - from) * ease(step / steps);
}

And then we "just" apply this function to all coordinates and alpha values for as many frames between keyframes as we like. This is not exactly rocket science, but somewhat fiddly and annoying. Simply refer to the Gist, copy the code and adjust it to your liking.

Now you can run the main script and bask in the glory that is a tiny animated JSON snippet:

Next steps

Once you have the rendering pipeline working in general, you will probably want to tweak it to deliver what you actually need:

• To adjust the look and feel of your animation check out the documentation on styling! The script already contains an HTML scaffold - if you need more CSS, simple dump it straight in there.
• For video editing, it makes sense to skip ffmpeg and just save the PNG screenshots, drag them into your video editor's timeline, and go from there.
• If you want to show off code in a presentation using Keynote or PowerPoint, you can render each keyframe transition into its own tiny video. Dragging the videos into your presentation software should automatically create a slide for each video that auto-plays on activation (at least I have been told that it works like this). In case you use something web-based, sticking to the default HTML and CSS is probably the more reasonable option.

Let me know if you build something interesting based on this script!