Adaptive music: how game scores react to what you do

Why a game score cannot be written to a fixed timeline

In most games, how long a moment lasts depends on the player. A fight might take ten seconds or three minutes. One player may cross a forest quickly; another may stop to explore every corner. A piece of music written to a fixed length will either end too early or keep going after the moment has passed.

The simplest solution is a loop: a piece that repeats seamlessly for as long as the player stays in an area. Loops are still widely used, but they cannot react. If the player is suddenly attacked, a looping exploration theme carries on regardless. Adaptive music tries to solve that problem by building the score so that the game can change it while it plays.

There are two main approaches, usually called vertical layering and horizontal re-sequencing, and most modern games use a mixture of both.

The Layer Mixer elsewhere on this site lets you try the first of these yourself.

Vertical layering: adding and removing parts of one piece

In vertical layering, a single piece is recorded or produced as several separate parts, often called stems, that all play in sync. A typical set might include a soft pad, a bass line, a rhythmic pattern and percussion. The game fades layers in and out depending on what is happening.

When the player is exploring, perhaps only the pad and a light rhythmic figure play. When enemies appear, the bass and drums fade in. When the danger passes, they fade out again. Because every layer is in the same key and tempo and runs in parallel, changes can happen at any moment without an awkward jump.

The Layer Mixer on this site demonstrates exactly this. It plays four layers synthesised in your browser, a pad, a bass, an arpeggio and drums, and its explore, danger and victory buttons change which layers you hear. Winifred Phillips, profiled here, is one of the composers best known for writing modular, layered scores.

Horizontal re-sequencing: choosing the next section on the fly

Horizontal re-sequencing works along the timeline rather than across it. The score is written as a set of short sections, such as an introduction, several loops at different intensities, transition bars and endings. The game decides which section to play next based on what is happening.

For example, a combat track might keep cycling between two loops while a fight continues, move to a more intense loop when the player's health drops, and jump to an ending phrase when the last enemy is defeated. To avoid jarring cuts, changes are usually timed to happen at the end of a bar or phrase.

The music for Bungie's Halo games, discussed on our Martin O'Donnell page, was written in sections that the game could start, loop and end depending on combat. Grant Kirkhope's Banjo-Kazooie used a related idea, crossfading between different arrangements of the same melody as the player moved between areas.

Stingers, transitions and the small pieces that join everything up

Alongside the main music, adaptive scores use short cues:

• Stingers are brief musical phrases played over the top of the current music to mark an event, such as finding a secret or landing a final blow.
• Transitions are short bridges written to move smoothly from one piece or key to another.
• Endings or tails let a loop finish naturally instead of stopping abruptly.

These pieces are small, but they do a lot of the work of making a score feel as though it was written to the player's exact actions.

iMUSE: an early adaptive system from the early 1990s

One of the best-known early systems for interactive music was iMUSE, developed at LucasArts by composers Michael Land and Peter McConnell. It was first used in Monkey Island 2: LeChuck's Revenge (1991), and LucasArts patented the system in 1994 (US patent 5,315,057).

iMUSE worked with sequenced MIDI music. It allowed the score to wait for a suitable musical point, then switch to a new theme or insert a transition, so that music changed smoothly as the player moved between locations. It worked a little like a pit orchestra in a theatre, extending or shortening a passage until the action on stage was ready.

Land started work on the system after becoming frustrated with the limits of the audio tools available when he composed for The Secret of Monkey Island. Many ideas now standard in game audio, such as transitions that wait for the next musical bar, can be seen in iMUSE.

FMOD, Wwise and the middleware composers use today

Most modern games use audio middleware: software that sits between the composer's audio files and the game engine. The two most widely used are FMOD, made by Firelight Technologies and first released in 1995, and Wwise, made by Audiokinetic.

In these tools, a sound designer or composer sets up the rules for how music should behave. They might define states (exploring, combat, victory), parameters that change continuously (such as an intensity value from 0 to 100) and the transition rules between pieces. The game then sends simple messages, such as setting intensity to 80, and the middleware handles the musical result.

Both tools integrate with major game engines including Unity and Unreal Engine, and both offer free licences for non-commercial use, which is useful if you want to learn. Our guide on getting into game music explains why learning one of them is a practical first step.

When adaptive music works well, and when it is noticed

Adaptive music is often most successful when the player does not consciously notice it. Changes that are well timed and musically natural simply make a game feel more responsive. Problems tend to be obvious: a combat theme that keeps starting and stopping as enemies move in and out of range, or a transition that lands in the middle of a phrase.

Composers and audio teams therefore spend a great deal of time tuning thresholds and timings, such as how long the game should wait before deciding a fight is over. Journey (2012), discussed on our Austin Wintory page, is an example where the music was planned together with sound design so that both responded to the player. For how these ideas developed into today's large productions, see our adaptive and modern era page.