What have I been doing for the past month?

I am writing this just because it has been about a month since my last development update (either in the form of youtube video or social media post). I have been extremely busy but unfortunately there is not exactly much I can show off right now and say "look at this cool thing I added" or anything like that. So as a warning there is probably not going to be anything particularly exciting here unless you have some deep interest in application development and persistent data structures.

After the latest build I really want to work on another bug-fixing update to keep trying to get things more stable but people weren't quite reporting new bugs quickly enough and I ended up working on other stuff. I will try to make sure to address all the bugs that have been reported in the meantime in the next build!

Usually I like to force myself to record development videos on youtube just to give an idea of what I'm up to, even if they are not immediately understandable to the person watching. The problem with that recently is I usually want to start a video by giving a brief explanation of what I'm doing, but what I'm doing right now is not very easy to explain in words especially with my slow brain, probable aspergers and speech impediment, so I was turning on OBS and then wasting half an hour trying to explain and then giving up and turning the recording off. I think I did that about 5 times so that adds up to a bunch of wasted development time.

What I am basically doing is transitioning a large part of the business logic from a data-oriented design over to a Bolivarian style value-oriented design.

Data-oriented design is a way of structuring things which is very popular in game development and it has served me well for a long time but I realised it's not going to work anymore going forward.

"Value-oriented design" is a more nebulous phrase which vaguely refers to structuring the high-level logic and state of your application as value types which can be efficiently copied around.

"Bolivarian" is a word I just invented 2 minutes ago and is a reference to an ex-Ableton engineer named Juan Pedro Bolívar who has given several very interesting talks about interactive application programming in C++.

I'm not sure how much of an impact these talks have had on the industry at large but they have had a huge impact on myself and my approach to development.

At its heart these ideas rely a lot on a technology known as "persistent data structures" (a.k.a. "immutable data structures") which has no language-level support in C++ but luckily Mr. Bolivar also wrote a very nice C++ library for working with them.  It's quite an awkward library to use but I think it's probably the best you can possibly do without actual language-level support for persistent data structures.

Persistent data structures are not new to Blockhead, I've been using them all over the place for a long time now in order to achieve various complicated things. However I never made the transition to representing the main project state as one big high-level persistent data structure. I did try to make this transition a couple of years ago but I just couldn't figure out how to do it.

Why am I doing this?

This all stems from an idea I implemented about 1 month ago where every time you select a bunch of blocks and use the Bounce function to create a new sample from it, all of the information about that configuration of blocks is stored with the sample, so even if the original blocks are deleted, you can restore them again from the sample:

https://x.com/ColugoMusic/status/1898022125195149533

I implemented this and it worked fine. The big issue is that the persistent data structure (PDS from now on) being used to represent the bounce, was being constructed from scratch rather than being derived from an existing PDS.

This is basically how I have been doing a lot of things for a long time, mostly anything involving historic or future versions of project state. The undo/redo system made heavy use of this, for example. Whenever I needed some kind of snapshot of the high-level project state, I would just construct a PDS by reading data from my data-oriented application state (basically a big database). That PDS can then be efficiently passed around and used for whatever, but by constructing it that way we lose a huge amount of the benefit of using persistent data structures to represent things in the first place.

If I need some snapshot of the project state at a certain point in time, instead of constructing it by reading all the data from somewhere else, all I should need to do, if the application is written correctly, is take a copy of the PDS which already exists. This is more or less as efficient as copying a couple of pointers, and is vastly more memory-efficient due to the way persistent data structures work.

Unfortunately I couldn't just take a copy of a PDS which already exists because I wasn't using a PDS to represent the high-level state of the project. I was only using PDS's for "this and that, and this thing over here", and often constructing new PDS's on the fly for situations where they were needed.

What will be the result of these changes?

1. Much lower memory usage.

I probably won't bother actually measuring anything to see the before-and-after difference but by always deriving new PDS's from existing ones we take full benefit of the underlying "structural sharing" of PDS's which can lead to dramatically more optimal use of memory.

PDS's are a little spooky in that you can do pretty extreme things with them such as representing more data than would even be possible to store in memory using traditional data storage techniques. I haven't taken the leap to representing sample data as PDS's yet but that is also quite an interesting idea even though I'm not sure what possibilities it would open up.

2. Much simpler code.

I'm already seeing this benefit. There have been several times over the past few weeks where I have reflexively yelled the words "oh my god this is so much better" to the empty room.

Working with PDS's in C++ involves writing a lot of boilerplate, even when using the Bolivar's immer library, but once you write all the boilerplate and hide it in a big header file the higher-level code becomes a lot easier to understand and more maintainable in my experience.

3. Photoshop-style undo/redo history.

Up until now Blockhead has used an event-based undo/redo mechanism. This means that every time the user performs an undo/redo-able action, I generate an "undo" event which can be played back to rewind to the previous project state, and also the inverse "redo" event. Then when the user undo's and redo's we're just playing back those events.

This is a very common and very error-prone method of implementing undo/redo in applications. Persistent data structures shit all over it. Once you are representing your project state as a PDS you can simply make a copy of the entire project state at any point in time. Then you can provide an interface to just jump backwards and forwards to any point. This also uses less memory than storing all those undo/redo events.

I'm not sure what this kind of undo history feature is actually called so I'm calling it "Photoshop-style" because I think a lot of people will understand that. It turns out many DAWs already have this style of history which tells me they are most likely using persistent data structures under the hood.

4. Easier implementation for other potential features.

Once all this is in-place certain things just become trivial to implement. This "reversible bouncing" thing for example caused me bunch of headaches when I implemented before, but I've already been able to delete that code and replace it with something very straightforward which achieves the same thing and with less memory usage.

I'm not sure yet what other doors are being opened up here but anything involving "historic" versions of project state should be much simpler to implement. As well as having the project-wide undo/redo history, there could also be a per-block history and/or the ability to "bookmark" a block's state at certain points in time. How useful would that be? I don't really know but I know that it would be easy to implement.

Photoshop also has the "history brush" which allows you to select a point in history and paint old pixels over the current ones. I'm not sure if there is any analogue to that in music software but it just gives an idea of the kind of weird stuff you can do with this kind of PDS backend.