Banjo-Kazooie, and the era of N64 MIDI
There’s one particular, classic example that is often used in classes and lectures about adaptive music, to demonstrate effective context-sensitive soundtracks: the instrumental fading and transitioning in the N64 game Banjo-Kazooie.
Rare’s 1998 platformer had the unenviable task of competing with the revolutionary Mario 64, and composer Grant Kirkhope took on the challenge of producing a charismatic score that could stand shoulder to shoulder with the melodic, catchy style of Nintendo’s game. And while Mario games are known for their industry-leading adaptive music techniques and melodic sound effects, Rare arguably went a step further with their use of audio techniques such as the “gibberish” dialogue audio, and their use of variable mixes within the soundtrack.
Variable mixes, made popular by LucasArts games such as Monkey Island, are a way to tailor the music to reflect changes in the game state, or to the game’s environments. When a player moves from section to section, and crosses various invisible lines in the scenery, the content of the music remains relatively consistent, but the arrangement changes- certain instruments fade out and are replaced by others, and the style is altered completely while remaining seamless and without breaking the flow of the melodies.
Skip to 10:40 for some good examples of layering between different variations
It’s a technique often referred to today as “vertical layering” or stacking, because the modern equivalent involves layering multiple versions/variations of the same theme or section of music on top of each other, to be played simultaneously- but with only one (or some) of them being audible at once. Then the game engine can fade the different versions in and out at appropriate times to smoothly transition between different arrangements.
It highlights one of the biggest changes to affect game audio in the late 1990s. Consoles such as the N64 used a MIDI-type system; the soundtrack’s ‘instruments’ sat in a library, to be triggered by a set of instructions, which specified which note to play, which instrument to play it with, the length of the note, and any extra information such as pitch-bends or special effects. This enabled the console’s sound chip to play the music ‘live’ much like a synthesiser or sampler would. And it enabled the variable mixing of games such as Banjo-Kazooie without drawing too much processing power.
This approach changed dramatically with the introduction of CDs and Redbook audio, as consoles like the Playstation allowed developers to stream music straight from the disc (through a very small RAM buffer). This enabled composers to use far more sophisticated sample libraries and live recordings, ushering in an age of cinematic, orchestral scores along with licensed music from well known artists. But new technical limitations related to storage issues and seek/read times made the use of adaptive music much harder within these new types of soundtrack- instead of ‘playing’ the instruments and fading from one to another, the simultaneous stacking of WAVs was required, which was next to impossible to process (even with huge amounts of compression and sample rate reduction). It wasn’t until much later, when larger buffers and other technical solutions were available, that vertical stacking and variable mixes could be easily applied to WAV based soundtracks. The introduction of middleware such as Wwise and fMod made these techniques more accessible for composers, and the move towards streaming data to RAM from hard disks, (rather than from media) solved the problem of load/seek times via instant-access.
So although we now have fully adaptive soundtracks using sample/WAV based audio, the problems thrown up by the shift in tech did seem to hinder the use of highly-interactive scores for a number of years, and represented a road bump in terms of innovation. Some would argue that we still aren’t seeing much more use of adaptive music now than we did in the days of cartridges and MIDI. When I asked Grant about it he seemed to agree:
“I do feel that we’re now absolutely back to the same point where MIDI was back then. And I could be wrong but I don’t think I’ve heard anything as interactive as the stuff we did back then with Banjo or the Monkey Island days- maybe people don’t want to do it?”
He suggested the lack of innovation might be related to genre, and the increasing subtlety of these techniques:
Grant Kirkhope, Banjo-Kazooie’s composer
“It’s down to the game makers themselves, and the audio directors- people are using these tools that do all the fading in and out- but it’s maybe not as obvious. It’s like adding extra melody lines or whatever on to what’s already going on underneath- I guess you might not notice that as much, unless you’re really paying attention. I know for instance the Call of Duty guys are doing this stuff- but I guess when I did it it was really obvious- e.g. cross the line and the music changes to another instrumentation. If you’re looking at the shooter games, the FPS games, maybe people don’t need it as much- apart from jumping in and out of combat- perhaps the platforming genre lends itself to that very well- whereas perhaps a first person shooter maybe doesn’t.”
It’s hard to deny that the abstract, brightly coloured worlds of Nintendo games lend themselves more to overtly interactive soundtracks than the realistic, cinematic shooters and adventure games that are more popular today. And it raises the argument that music which is too subtle with its adaptive implementation risks going un-noticed; sometimes a jarring switch form one type of music to another can benefit the player more than a transition whose purpose might not have any conscious effect on the player. Changing states from ‘exploration’ to ‘battle’ and back again is one of the most widely used techniques, and that change has to be clearly signalled to the player for it to work properly.
And while these techniques remain in the background in the more mainstream blockbusters, there’s now a huge wave of innovation within the world of indie gaming- where composers and sound designers are free to experiment and bring these techniques to the fore. As a result, we are seeing musical and rhythmical concepts take centre stage as gaming mechanics, and entirely new genres of game are quickly becoming standard, albeit outside of the AAA scene. But it’s important to remember how influential the original uses of MIDI fading really were, and what a powerful effect it had on sparking the imaginations of composers at the time.
Many thanks to Grant for chatting with me! You can find out more at his website or wikipedia