Raytraced Audio Update - New Features and New Approaches

Patreon limits videos to 8 Mbps. The full resolution video and 65 screenshots are available here

Sector's Edge is now using the latest Raytraced Audio system and it sounds incredible. There was no nice way to do this, I had to delete all the old audio code from Sector's Edge and connect it to the new system, one function at a time. My GitHub stats show 6071 lines added and 21519 lines deleted.

The environment is fully destructible + buildable, so you can test out different room shapes and sizes. You'll also be able to place sound sources around the map and trigger them via the UI or keyboard shortcuts.

In the bottom left you'll be able to see:

• wind ambience direction

• reverb stats - the diagram shows the size of the room (blue box), and how much energy is returning back to you (how full the box is)

• how 'outside' you are

• update time - how long it takes to re-trace all 960 rays and update reverb / ambience / muffling / permeation stats for all sounds

You can press escape and hover over active sounds to view graphs for:

• LOS (green) - line-of-sight muffle strength

• PER (orange) - permeation energy

You can also click the X to delete the sound, or click the lock icon to keep the graphs visible.

Patrons will get access to this demo first, but there's still a few more things to do:

• ability to change maps and weather

• improve ambience directionality - ability to hear rain from multiple directions

• fly camera

• setup sound sources and trigger them

As I'm working on this, I'm thinking of ways to improve (or completely replace) this raytracing system. I've outlined them below:

Ray Sets

There are some environments where the majority of the rays stay near the player rather than travelling outwards to discover more parts of the world:

One option to improve this is to use the results of the last raytrace to determine the 'best' and 'worst' rays, in terms of data gathered. For example if two rays are fired in a similar direction and they both discover the same sound sources and produce the same reverb information, then the 2nd ray is obsolete. The first ray should be given 2x the influence (since its doing the work of 2 rays), and the other ray should be fired in another direction.

The other option is to slightly randomise the reflection angle of each ray, and then over time pick the direction that caused the ray to cover the most distance.

Both options would require dividing the rays into two sets:

• the 'base rays', these never change to ensure all directions are still checked

• the 'supporting rays', which explore different paths based on the heuristics gathered by the base rays

As the environment around the player changes, the rays would then automatically adapt to cover the most ground possible.

Voxel Flood Fill

It's relatively expensive to cast green line-of-sight rays to calculate how muffled each sound source should be. With more active sound sources, more rays need to be cast, and the system runs slower.

One approach I've considered for a long time is to group sound sources together if:

• they have line-of-sight with each other

• they are close to each other

For example when firing a minigun, each bullet makes a sound when it hits a surface. Each bullet might collide with a different surface (the minigun is inaccurate), and each sound must be raytraced to determine how muffled it is.

I'm already grouping sounds together per-player, for example footsteps, gunfire and proximity chat all share the same raytracing stats because they originate from the same player.

But to improve this further, I'm considering a new flood fill approach, which is completely separate to the raytracing approach. Both approaches will be supported, so you can choose the best one for your game.

Each sound would be positioned on the voxel grid, and the flood fill would expand outwards from your player in search of them.

Each time it steps outwards from your player, the 'distance' number increases. When a sound is discovered, the current distance number is compared against the straight-line distance to the sound. In the diagram above, the green sound is 5 units away from your player, and the distance number is 5, and 5 / 5 = 100% clarity so the sound is not muffled.

But if there's something between you and the sound, it'll take more steps to reach it:

In this case it took 9 steps to reach the sound source, so it should be slightly muffled. The more steps it takes, the more muffled the sound should be.

This system would require tweaking, as the sound above shouldn't only be 5 / 9 = 55% clear, as that's a pretty extreme reduction given it's only a small wall between you and the sound. Here's a few more examples:

The same reverb stats as the raytracing approach (returning ray percent, returning ray length) can be calculated in this approach too. Every time the flood fill finds a non-empty block, it would check the current distance number against the straight-line distance from your player. If they're the same, then there's line of sight with your player, and the usual reverb stats can be calculated from that.

I'm not sure how permeation would work with this. Maybe a separate close-range flood fill could be used to calculate how much 'mass' there is between you and each sound source. I'm also not sure how ambience would work, maybe raycasting is needed for that.

In theory this approach should run faster, because less voxels are visited and we don't waste time trying to find sounds that are clearly enclosed in a separate room.

Vector Fields

Another potential improvement for the raytracing approach is to use vector fields to 'guide' rays away from dead ends.

The vector field will start empty (each point in the field would have no direction), and as a ray travels throughout the world, it'll alter/push the vector field direction to either the left or right. The next ray that travels along the same part of the vector field would then be slightly pushed to the side.

This is all guesswork - I'd have to run a simulation - but my theory is that over time the rays will spread away from commonly visited areas, and be forced to explore the path less travelled. This might even cause them to bend around corners.

Another approach would be to produce the vector field from the voxel data. Empty space would have no direction, and voxels would point towards the empty space, causing rays to be repelled like a magnet away from walls. This might just be a complicated version of the flood fill approach, but I'd like to test it anyway and see how it goes.

Miscellaneous

I faced a few strange rendering issues with textures when trying to upgrade SE to the latest Vercidium Engine version. The worst issues was flickering textures in the distance, as if they had no mipmaps. The problem ended up being the normal textures, but while I was debugging I ended up with this strange edge-detection effect. Not sure how it works but it might be useful for effects later on!

I also fixed a memory leak (the game allocated 300,000 buffers for indirect rendering) and cleaned up a lot of other messy engine code. I'll merge that into the main branch once things are stable.

So far 1156 people have filled out the https://vercidium.com/audio form! I'll share the survey results here soon.

High-res images and videos for this post are available here