The systems behind the sound & music of Vessel

Kieran Lord
Strange Loop Games

About Me

• Working at AiGameDev.com
• Previously worked on...
• Destroy All Humans 2 - Pandemic Studios
• Engine Technology - Krome Studios
• A bunch of shelved next gen titles :(
• Alternator - Defiant Development
• VESSEL - Strangeloop games

Agenda

1. Dynamic Sounds
2. Vessel Music System
3. Music After Vessel

About VESSEL

• Indie puzzle platformer
• PC (on steam) with Mac & Linux coming very soon
• XBLA & PSN coming soon
• It's in the humble bundle right now!

About VESSEL

• Centers around water physics and characters made out of simulated water
• Everything in the game works off physics, even button and levers
• So we needed to make the sound & music react dynamically too!

About VESSEL

Dynamic Sounds

Dynamic Sounds

Parameterized sounds - FMOD, WWise etc

• Sound events with parameters.
• Start or stop an event and set the parameters (floats) from gameplay.
• What actually happens is controlled by awesome tools for a sound engineer

Dynamic Sounds

Parameterized sounds - Pathfinding fluro example

• One of our debug values from pathfinding "re-normalized distance to goal" actually used as a sound parameter
• The value ranges from 0 (the start location) to 1 (arrived at goal) and never decreases.

Dynamic Sounds

Parameterized sounds - Pathfinding fluro example

• It actually worked great as a debug tool, an audio pathfinding visualization
• More importantly adding one parameter added a huge amount of life to the characters

Parameterized sounds - Pathfinding fluro example

Dynamic Sounds

Parameterized sounds - FMOD, WWise etc

• Events can control multiple playbacks, layered sounds, effects.
• Completely configurable by a sound designer in a separate editor
• Don't just use them to play a static sound!

Dynamic Sounds

Rushing water - Drop clusters

• Static sounds for water are only acceptable if you have static water.
• All water in vessel is clustered into circular groups of drops provided they have a similar behaviour

Dynamic Sounds

Rushing water - Drop clusters

• Used by many game systems to interact with the water simulation.
• The average density, velocity and the number of drops in a cluster are used as parameters to the "Water rushing" sound.

Rushing Water

Dynamic Sounds

Rushing water - Output parameters

• The water rushing sound only takes 3 parameters:
• Speed
• Drop count
• Density

Rushing Water

Dynamic Sounds

Rushing water - Clustering drops

• Main drop clustering algorithm (used to access and manage groups of water)
• Large enough groups of water drops of the same liquid type, and similar behavior are clustered into circles

Dynamic Sounds

Rushing water - Clustering drops

• Each circle then calculates aggregate/average information about the water there:
  • drop count
  • pressure
  • velocity
  • attached to character id
• Each circle then calculates average information about the water there

Dynamic Sounds

Rushing water - Assigning & Removing playbacks

• Each update we sort the clusters based on size, then filter them based on...
• Is the cluster on screen, minimum movement speed and minimum drop count
• The 5 top ranked clusters of each liquid type will be allowed to start sounds (if they're not playing already)
• All other clusters must cue out their sounds (again, unless already doing so)

Splashing Water

Dynamic Sounds

Splashing water - Output parameters

• The water collision sound only takes 2 parameters:
• Collision Count
• Percent Hitting Player

Dynamic Sounds

Splashing water - About

• Dynamic water needs to create sound to mimic the simulation's behavior using as few sound playbacks as possible.
• We cluster collision events, and use each cluster as a sound playback

Dynamic Sounds

Splashing water - Clustering collisions

• Every water collision with a rigidbody is stored for about 0.15 seconds
• If that collision occurs within an existing cluster's circle, the collision is added to that cluster (if the drop matches the water type of that cluster)
• If there is no cluster where a collision occurs, a new one is created

Dynamic Sounds

Splashing water - Clustering collisions

• The size of a cluster's circle is proportional to the number of collisions it stores, plus a starting radius
• When a cluster has no more collisions, it's destroyed

Splashing Water

Dynamic Sounds

Splashing water - Output parameters

• Each cluster is an audio playback with parameters
• The water type of the cluster dictates which sound we play (water, lava, red goo, blue goo, etc)
• We pass the rest of the parameters onto FMod to adjust the sound playback
• The parameters would need to be smoothed, but fortunately FMod handled that for us

Adaptive Music In VESSEL

Why you should care about adaptive music

• Players are trained to recognize visual and sound cues, not music ones
• You can use music to guide player behaviour, with them only being partly aware of it
• It's an underused technique in games with only makes it more effective when done right

Vessel Music System

Why you should care about adaptive music

Music Layers

• Vessel had only 5 layers per song, remixed from the instrument recordings (stems)
• Each layer is always playing something interesting
• Fade layers in and out in different combinations to change the feel
• Timing switching between them is key

Vessel Layers & Stingers

Vessel Music Box Demo

Vessel Music System

Timing System
Motivation

• Snapping or crossfading between layer states is very limited
• Often want to bring layers in at different times and fade them out after time
• Music changes need to happen when it makes sense in the music

Vessel Music System

Timing System
Music Locations

• Absolute
• Relative to beat
• Relative to bar
• Relative to 'significant beat' marker

Music Locations

Timing System - DSAR!

• 
• Delay Attack Sustain Response envelope, volume vs time
• Linear series of states, each runs until a specified music location.
• Sustain is special: it can optionally run for a duration, or until interrupted

Vessel Music System

Music Boxes
Problem

• Needed a way to rig music to roughly 140 unique puzzles in the game without going mad.
• Robust solution to the player being able to walk between multiple puzzles at times
• Wanted the best control we could have over our five layers at all times

Vessel Music System

Solution!

• Having a great sound engineer (Leonard Paul)
• Each puzzle has a list of states it goes through in order to be solved
• By default, the music box isn't allowed to play a state lower than it's current

Vessel Music System

Solution!

• Each state is a DSAR timing envelope for each layer, stored in a shared asset (so multiple music boxes can use it)
• Music boxes can be connected to game logic by script, or by game actions

Vessel Music System

Music Boxes - ALL the music boxes (147 in total!)

• Took two weeks full time to hook these up to every puzzle in the game
• Very hard to test and maintain
• Totally worth it

Vessel Music System

Music Boxes - The presets

• A selection on which layers will play
• Timing information on how each layer fades in/out (more on this in the next section)
• Multiple music boxes share the same presets (even ones using different tracks)

Music after Vessel

Music after Vessel

Motivation

• Lots of ideas in vessel that we didn't use:
• Using music locations to trigger gameplay events
• A behavior tree approach to music boxes

Music after Vessel

Motivation

• I really want to see more people using adaptive music
• A lot of this stuff is a bit inaccessible to non-sound and non-AI people
• I use unity a lot and there's no available system for this in that engine

Music after Vessel

Goodbye music boxes

• Vessel's music system was very specific to the needs of the music we were working with...
• Music boxes could only store a linear array of states...
• Worked well for hooking up level events to music, but what about more dynamic events events?

Music after Vessel

Goodbye music boxes

• Player in a dangerous area?
• Time till support arrives?
• Combo multiplier at x10?
• We need a way to represent the behavior the music should have based on and between these events.

Music after Vessel

Hello music tree!

• It's a behavior tree of nested playslists and state selector nodes.
• Credit has to go to WWise's editor, as their system does something very similar to this
• Leaf nodes are playbacks (which can have multiple layers)

How a music tree works - Playlist nodes

Music after Vessel

How a music tree works - Playlist nodes

• Playlist nodes select a child to play
• They can be set to pick a child sequentially or randomly
• There is a setting to play a single item, default behavior is to play all items

How a music tree works - State nodes

Music after Vessel

How a music tree works - State nodes

• State nodes are a list of states, each with a name
• When an event that matches the name of one of the states is received, the state node will switch to playing that state
• This state switch also happens if the node isn't playing, (note: events are passed recursively to all nodes)
• This covers most gameplay to music selection, it's simple but very powerful

Music after Vessel

How a music tree works - Crossfading

• When switching from one playback to another, the new playback crossfades over the old one
• This works exactly the same as the DSAR system mentioned earlier, except the Release state is only used when fading to silence

Music after Vessel

How a music tree works - Crossfading crossfades!

• You might be wondering, what if I switch between playbacks twice... and the system is already crossfading?
• A good way to handle this is to treat crossfades as nested... eg. Crossfade(newest, Crossfade(older, oldest) )
• It's very easy to implement this!

Music Locations

Music after Vessel

How a music tree works - Crossfading Algorithm: Special cases

• There's a few special cases we need to cover:
• If x ever reaches zero, you can stop and remove all items from the current item to the end of the list
• To fade something in that's currently fading out, set it's "desired volume" to it's "real volume" and move it to the front of the list

Music after Vessel

How a music tree works - Synchronization and stealing

• Most of the time changing from one playback to the next requires to to sync up the timing
• Currently assume that tracks that play together are loops of the same length or multiples of each other.
• If the previous playback is already playing a layer that will be used in the the new playback, don't fade it out and in again...
• Steal it! (and keep it playing)

Music after Vessel

How a music tree works - Streaming

• Knowing what will play next is a tricky problem
• Refitting your system to know which track to stream next is very hard
• Best to design your system with this in mind

Music after Vessel

How a music tree works - Streaming

• Pre-calculating the next item to play seems like a simple solution, however...
• State nodes being able to change, and repeat counters on playlists create lots of edge cases
• It's very similar to a tree based planning problem
• A good approach may be to store the whole tree's state in a form that can be then stepped and manipulated
• (Haven't tried this yet though)

Wrapping up

• Chances are that you already have an incredibly powerful dynamic audio toolset linked into your game, use it!
• AI techniques like clustering are applicable to adding behavior to dynamic sounds and music
• Adaptive music is really powerful, and not that difficult to implement.
• Look outside characters to find systems that can use some of the AI love :)

Questions?