![Raytraced Audio - Draft V2 [8:40]](https://img5.samukata.com/storage/9/qc/hy/4e9b88-019e7176-873f-74c7-a04d-356ab0b6cb65.jpg)
Raytraced Audio - Draft V2 [8:40]
Added 2025-03-07 12:15:57 +0000 UTCThank you to everyone (Patrons, friends, family) who provided feedback on the the first draft. Here's the second draft, which addresses the feedback listed below.
I would love to hear your feedback on this version as well. It's a constant iterative process!
V2 Changelog:
The initial segment about spatialised audio has been cut down heaps. This is because the video is about raytracing, but the raytracing segment starts at 2:40, which is too far in. In this version the raytracing segment starts at 1:48, which is much better but I'd still like to cut it down
Added an explanation about why rays travel away from the player, rather than towards them (like sound in real life)
Many script adjustments to remove unnecessary sentences and streamline the video
Various animation timing improvements
Added "You" and "Jamie" labels above the people in the office intro
Replaced 'none' and 'max' with 'quiet' and 'loud' on the reverb cube
Window now appears correctly in the rain scene (and rain un-muffles correctly)
Thicker rays in the rain scene (easier to see on mobile)
Brighter world for the permeation segment
Changed ray colour to orange in the permeation segment
Added a sentence about dot shapes in the deaf segment
Still on my todo list:
Add text labels to better connect the visuals with what I'm talking about
At the end mention the demos are 2D, but in game it's actually 3D - tie it back to the intro where I say our brains can locate sounds in 3D
Replace music in the reverb segment with a simple beep sound so it's easier to hear the reverb changes. Pause between each sentence so the beep sound can be heard clearly
In the permeation segment, mention materials affect permeation and reverb as well, but I haven't explored that yet
Move the camera around more in the reverb segment to zoom in on what I'm talking about (pillars appearing inside the building, rays escaping outdoors)
Brighter / more interesting scene for the deaf segment
Polishing:
Stick my sound dampening panels on the walls, grab all the pillows and lounge cushions in the house, and record the final script
Update the timing of all animations to match the script - no freeze frames!
Add diagonal transitions between segments, like in my crepuscular ray video
Increase shadow resolution
Test recording in 4K
Thanks again to everyone who provided feedback!
Comments
> Are the raycasts just used to determine how the muffle, direction, and reverb parameters are changed, Yes that's right. I currently blend between a few reverb EFX presets (e.g. wood_smallroom, wooden_hall, valley). Materials are on my list but I need a better understanding of what all the reverb parameters do first. Plus a list of all the absorption/reflection frequencies of all the different materials. > Potentially you could controll the steepness of the filter by the material and thickness of the walls/objects? At 2:40 the muffle strength is a set constant. Permeation (thickness of walls) is introduced at 5:53. Materials are also on my list for permeation :) Good points about lowering volume rather than filtering. I've only ever used low pass filters, I haven't experimented with volume changes as well > So pretty much simulating how audio works in real life, like HRTFs Doesn't HRTF already simulate how sound waves bounce around the head/shoulders/ears? I assumed it already delayed/muffled sounds (among other things) based on the direction of the sound already. I couldn't find concrete sources on what HRTF actually does so I'm not sure. > I'd love to talk more about this type of stuff in detail, give feedback/ideas/learn/whatever else, I'm super super super interested in the realism side of audio. Same here! It's a very interesting topic and it's been fun to work on the raytracing side. Materials and more complex sound science are daunting for me, so it'll be nice to have some help :)
Vercidium
2025-03-14 11:16:05 +0000 UTCIm really looking forward to see this fully fleshed out in the game (and hopefully other games too). Are the raycasts just used to determine how the muffle, direction, and reverb parameters are changed, or does it actually create the reverb from scratch based on how the rays reflect based on the geometry like Microsoft's "Project Acoustic"? From the examples I heard in the video, I think the muffling is a little extreme, as in I think the filtering is too steep. Potentially you could controll the steepness of the filter by the material and thickness of the walls/objects? At the example at 2:40, I think that the sound should of course be lowered in volume the more occluded it is, but I don't think that it filters quite right. I think the sound should be filtered heavily if it is 100% occluded, but if there is still an opening between the sound and the listener, I think it would be more natural for it to be lowered in volume, but filtered out less than the example was. And when the sound is completely occluded, then start more agressive filtering. I don't know where this came from, but I've been nerding out over what makes audio sound real for a few months now, so it's awesome to see this type of stuff being worked on and being a priority rather than just some thing you're required to do in a game. (Literally been shaking sugar packets around my head to figure out what is happeneing to make it sound the way it does lmao im going insame) I've been thinking about an idea for "realistic" audio where both ears are considered different objects and are placed accordingly related to the head, so when audio is played, it is delayed depending on how far away the sound is from each ear, and using more complex filters to "muffle" the sound according to the direction of the sound to the ears. So pretty much simulating how audio works in real life, like HRTFs but way more complicated and hopefully better sounding. I've been trying to do something like this in Ableton Live (because its the software I know best) by controling distance and direction with effects like eq and reverb. Its a fantastic peice of software, but it just isnt made for this type of thing. Also, if you're open to it, I'd love to talk more about this type of stuff in detail, give feedback/ideas/learn/whatever else, I'm super super super interested in the realism side of audio.
LilRoosta35
2025-03-13 00:50:30 +0000 UTCGreat questions - it’s raymarching against a voxelised version of the scene. It’s not a SDF, just regular chunks of 32x32x32 voxels. The same format is used in my voxel optimisation video: https://youtu.be/40JzyaOYJeY I haven’t experimented with compute shaders. The full raymarching function with all this audio functionality is over 1000 lines long. I’m not sure if that complexity would work well on compute shaders or if I’d need to re-architect it For these visualisations the full path of each ray is stored in a buffer and rendered as simple lines, with a fancy shader to make them fade out. This creates the illusion that the rays are moving :) This buffer is mapped persistently so the background thread can write to it without worrying by about flushing / mapping / unmapping the buffer I’m not sure how much control Godot gives you of underlying buffers but this visualisation is pretty primitive and could be replicated using what Godot offers. All the raytracing itself - as well as OpenAL - is on the CPU so it would work well with Godot or any engine
Vercidium
2025-03-12 13:15:36 +0000 UTCI was looking into ways to raytrace audio like this when I found that serendipitously you were working on just that. Is this method using regular raytracing or is it raymarching against the voxelised scene as an SDF? I was thinking in order for it to be efficient it would need to be run as a compute shader, so I was surprised to find the raytracing is run CPU-side. Especially for visualisation where you want to get the raytraced results onto the GPU. I had no idea whether that would create a significant bottleneck or not. Admittedly I'm naive when it comes to all this, lacking practical experience, so I'm very interested in the technical details, though that's probably a bit too specific for a general audience for a YouTube video. I'm currently using Godot (again, admittedly a bit new), but if possible I'd love to have a go at testing/implementing your solution in the engine, if it's within my capabilities to do so (and if not, I'll learn).
D W
2025-03-12 02:24:24 +0000 UTCThe audio raytracing code isn't on GitHub yet. I need to figure out what kind of product it will be and how I'll licence it first Blending between LODs smoothly like that is a topic I'm interested in but haven't dived into yet. I wonder if the additional info in each vertex - that's required to blend between the two - would make the models a lot slower to render. I'm keen to experiment with it though
Vercidium
2025-03-09 09:16:32 +0000 UTCHow can I see the code for the audio raytracing? Is it in the ve engine GitHub?
J19
2025-03-09 02:06:52 +0000 UTCAnother idea I had when I saw Runevision's dithering video is an automatic LOD system for meshes that changes the number of vertices on a mesh (non-uniform density across a single mesh) in real-time so that you never see pop in. Do you think something like that could work? (Placing quads that connect up together on a surface is kind of similar to placing pixels and texture patches)
J19
2025-03-09 00:09:39 +0000 UTCI haven't worked with Godot before (or any game engine really!) so I would greatly appreciate your help. I watched Runevision's dithering videos recently and never considered that approach for the sound visualisation! That would work really well. I have a way to map from voxel back to triangle space, but wasn't sure how best to display the dots on a triangulated model. This dithering approach would be perfect
Vercidium
2025-03-08 22:35:47 +0000 UTCGreat point, this visualisation wouldn't work there. The traditional approach of arrow indicators or highlights on the edge of the screen would be much better. With no environment to block your vision, everything in your field of view wouldn't need any extra indicators. Anything behind/below/above/left/right would need a directional indicator Mapping the direction of sounds to a sphere around your player would work for a third person game but I'm not sure how it would work for a first person game. Is that what you mean? I'm aiming for a more immersive solution where the sound visualisation is part of the game itself (i.e. painted on the surfaces around you).
Vercidium
2025-03-08 22:32:53 +0000 UTCWith the current deaf audio visualization, what if there are no walls to paint the sound on? E.g. in a game where you're high up in the air on a flying mount? Wouldn't it be nice if it was just directly mapped to screen? That would aid with immersion because they wouldn't have to think about where to look for sound. And it would help with understanding direction even more because the mapping of pixels to direction would have higher resolution that wouldn't be dependent on an additional incident angle to the surface.
J19
2025-03-08 16:06:12 +0000 UTCWow! I love it! I'd love to help work on a Godot implementation (preferably a GDExtension?). I'm tempted to start working on one for personal projects right away. Also on the deaf sound visualization maybe it could be nice if it disabled other shaders that tint or color the player's perception and then it became a smooth gradient of color that colored the whole screen depending on what is supposed to be heard there. (So that they can clearly see that one spot of the screen has gunshots while another spot has footsteps) You could also color band by frequency or/and use icons to label sounds (that could be customized by the game developers) or utilize structured techniques like stable dithering to convey information. (Stable dithering: https://youtu.be/HPqGaIMVuLs?si=ejd5wxo1aS1EFBja https://youtu.be/EzjWBmhO_1E?si=0khWbcxqDIpUnZLz )
J19
2025-03-08 15:52:30 +0000 UTC
