Why Neural Acoustics Modeling Is the Future of Physical Modeling Synthesis

Most virtual instruments either play back recordings of a real instrument or build sounds from a series of oscillators and other synthesis tools.

Physical modeling synthesis takes a different approach by trying to recreate the physical behavior behind the sound. Dreamtonics’ Neural Acoustics Modeling builds on that idea by combining the realism of captured acoustic performances with the kind of continuous control you’d expect from a modeled instrument.

Let’s take a look at what physical modeling is and where it came from, then look at how Dreamtonics is taking it in a new direction with Neural Acoustics Modeling in Instrument X.

What is physical modeling synthesis?

In its simplest definition, physical modeling synthesis is a method of generating sound by mathematically simulating how a physical instrument or acoustic system behaves.

Instead of playing back a recording, the synthesizer models things like the way a string vibrates, the way the body of an instrument resonates, the way a bow hits a cello, or the way breath moves through a trumpet.

Take a violin. The bow gets the string moving. The string sends energy through the bridge. The body resonates, and that vibration moves through the air to the listener.

A physical model digitizes some or all of that chain mathematically.

Once the system exists as a model, you can also change properties or combine parts that would be impossible to build physically.

How physical modeling differs from sampling and subtractive synthesis

Sampling begins with the result.

A player performs a note, the performance is recorded, and software plays it back from MIDI. Modern libraries may add dynamic layers, round robins, articulations, legato transitions, and multiple mic positions.

Subtractive synthesis starts with a harmonically rich sound, like a saw or square wave, then uses filters, envelopes, and modulation parameters to mold it into the final sound. It isn’t meant to understand the physical properties of a violin or saxophone.

Physical modeling focuses on the system that produces the sound.

I like to think of it this way:

If I were building a sample-based instrument, I’d ask, “What does this instrument sound like?”

But if I was developing a physical modeling VST, I’d instead ask, “What makes it sound that way?”

What physical modeling already does well today

Physical modeling already works really well with certain sounds.

It’s great for plucked and struck strings, along with electric pianos, resonators, mallet sounds, and wind or reed instruments.

Applied Acoustics Systems has several modeled string and electric piano VSTs.

The big advantage is that these instruments can react to how you play them.

Play harder with AAS’ Lounge Lizard EP-4 electric piano, and the modeled hammer strikes the virtual tine with more force, changing the attack and tone of the note rather than simply turning it up.

A short history of physical modeling

Long before AAS began developing modeling software, researchers were already experimenting with ways to recreate the behavior of acoustic instruments digitally.

In 1983, Kevin Karplus and Alex Strong published the Karplus-Strong algorithm, which showed that a short burst of noise sent through a delay-and-feedback system could produce a surprisingly convincing plucked-string sound.

That basic idea opened the door to more sophisticated ways of modeling how sound moves through an instrument. Researchers later developed digital waveguide synthesis to simulate waves traveling through strings and tubes, and Yamaha eventually brought those ideas into a commercial instrument with the VL1 in 1993

The VL1 became a landmark physical modeling synth. It modeled wind and string behavior, and responded to expressive performance controls. More instrument developers would follow suit over the next decade, carrying the same idea forward as computers became more powerful.

Where classic physical modeling falls short

The idea is elegant. But the real world is, well, complicated.

There are a lot of variables that go into the sound of a player bowing a violin. Small changes in bow pressure and speed. The friction of that bow repeatedly gripping and releasing across the string. The way the body is resonating, relevant to the room. How all of those pieces interact with one another.

Wind instruments can be even more complex, with added resonance and airflow variables.

Every extra layer = more computation.

Yamaha ran into that issue during its early Virtual Acoustic research. When its engineers tried to model cone-shaped wind instruments like saxophones, the math was simply too demanding for the DSP chips available at the time. To make the system work in real time, they had to simplify the model by approximating the cone with a series of cylindrical sections.

But even though the VL1 was incredibly expressive, it could also be demanding to play. Musicians had to control several different expressions at once, like breath pressure, pitch, and tone, all while playing the keyboard, which gave the instrument a steeper learning curve.

Developers eventually realized modeling the instrument was only half the job. They also had to account for the room, the reflections, and the way the sound reaches the microphones, which is where Neural Acoustics Modeling came into play.

What is Neural Acoustics Modeling?

Here at Dreamtonics, we use the term Neural Acoustics Modeling for our approach to recreating the behavior of real instruments and the spaces they were recorded in using trained neural models and measured acoustic data.

It’s different from both sample playback and classic physical modeling.

Traditional sample playback stores recorded performances and recalls them later. A traditional physical model works from mathematical descriptions of the instrument’s behavior.

Neural Acoustics Modeling learns from real performance recordings and acoustic measurements so the model can recreate that behavior without just triggering a bank of prerecorded notes.

The “neural” part can make it sound like generative music software, but that isn’t the case. Our VSTs don’t write melodies or compose parts for you. You supply the notes, while the neural system handles how those notes are performed and how the instrument behaves acoustically.

You’ve heard this style of modeling in Synthesizer V Studio 2 Pro. Instrument X takes that same kind of research and applies it to acoustic instruments.

How Neural Acoustics Modeling works in Instrument X

Training on multi-microphone session recordings

Each Instrument X expansion is designed from recording sessions with professional performers, each captured through a series of 8 to 11 microphones.

Those microphones cover close spots, a Decca Tree, wider positions, and room ambience.

Each one hears a different balance of direct sound, reflections, and decay.

But rather than compiling those recordings into a sample library for players to trigger note by note, the session data is used to train the model.

The dodecahedral speaker array

The Dreamtonics team also measures the recording space itself using a custom, patent-pending dodecahedral speaker array.

A dodecahedral loudspeaker is a 12-sided system used in room-acoustics measurement to approximate an omnidirectional source. Put simply, it sends sound around the room so engineers can measure how the space responds.

Research published in the Journal of the Acoustical Society of America has examined dodecahedral loudspeakers for room impulse-response measurement. They aren’t perfectly omnidirectional at every frequency, but they are very useful for measuring room behavior.

This speaker array is used to measure the acoustic paths between the source (the instrument) and the microphone positions.

It’s kind of like taking an acoustic fingerprint of the studio.

Separating the room from the instrument

Put a microphone up to capture a violin, and it’ll capture everything else in the room too.

Dreamtonics separates that capture using measured spatial data, which decouples the room’s directional acoustic response from the instrument source to create an impulse response. That’s what gives Instrument X more flexibility when it recreates the final performance.

With that information separated, Instrument X users can adjust the microphone perspective after they’ve generated the performance.

Holding phase coherence across mic feeds

One of the problems we run into using multiple microphones to record any source is phase.

If two microphones hear the same sound from different distances, the sound reaches them at slightly different times. When those signals are mixed, some frequencies can cancel each other out. Those little timing differences are why we get comb filtering.

Now imagine a neural instrument generating 8 to 11 microphone feeds at once. If those feeds drift relative to one another, you can get problems in the mix.

Instrument X uses its room measurements to keep every mic feed lined up with the others and in phase. That way, when you blend them together, they still sound like one instrument recorded in the same space instead of a bunch of separate signals fighting each other.

Neural Acoustics Modeling vs. sample libraries

The easiest way to see the difference between neural acoustics modeling and traditional sample playback is to put them side by side.

Area	Traditional Sample Playback	Instrument X / Neural Acoustics Modeling
Sound source	Prerecorded performances	Sound generated from a trained performance/acoustic model
Storage	Recorded samples stored locally	No large bank of triggered performance samples
Transitions	Recorded legato, scripting, or crossfades	Performance behavior can respond to the phrase
Dynamics	Often recorded layers + MIDI control	Dynamics can continuously change timbre and harmonic response
Articulations	Commonly patches, keyswitches, or articulation maps	Compatible articulations can stack on the same note
Mic control	Blending recorded microphone positions	Generate mic perspectives informed by measured room data
Loading	Samples may preload into RAM and stream from disk	No large sample-streaming workflow
Processing	RAM, disk, and CPU needs vary	Neural rendering runs locally without a GPU

Kontakt is a good example of the traditional sample-based approach.

Its Direct From Disk system keeps part of each sample in RAM and streams the rest from your drive as you play. Instrument X skips that sample-streaming model entirely because it isn’t pulling from a large bank of recorded notes, which is worth noting if you’re looking for Kontakt alternatives.

What it means for composers and music producers

So what changes when you’re actually making music?

No multi-gigabyte sample streaming

Large sample libraries can take up tons of storage and constantly stream recordings from disk, which can eat into your SSD space and make it harder to run heavy sessions. Nothing kills inspiration faster than having to wait for an instrument to load up.

Instrument X doesn’t rely on a large bank of performance samples during playback, so there is less dependence on loading and streaming audio as you move between pitches, dynamics, and articulations.

Runs locally without a GPU or constant internet connection

Instrument X’s neural engine runs locally and doesn’t require a GPU. You also don’t need a persistent internet connection while you work (assuming you’ve already activated your license).

That kind of local workflow depends on the software being able to render neural audio quickly enough on your own computer, which is why Dreamtonics puts so much emphasis on live rendering.

Continuous expression instead of fixed velocity layers

As a violin moves from ppp to fff, it doesn’t just get louder. Its harmonic response and tone change too. The whole performance intensifies.

Instrument X lets you move through those changes continuously instead of jumping between fixed recorded velocity layers. You can hear the difference in our guide to [programming realistic MIDI strings].

Is computational acoustics the future of virtual instruments?

Computational acoustics probably won’t replace traditional sample playback anytime soon. It doesn’t necessarily need to. If you already get the sound you want from a library, great!

What learned acoustic models add is another way to design instruments that respond more fluidly to a performance rather than relying on a fixed set of recorded results.

For musicians who want more control over how instruments behave, it can be a major payoff.

Hear Neural Acoustics Modeling for yourself

Physical modeling synthesis has been chasing the idea of building a more responsive system for decades.

Neural Acoustics Modeling gives musicians another approach to making music in the digital domain, and Instrument X is a prime example of that.

Hear it for yourself! Try Instrument X and explore the available instrument expansions in the Dreamtonics Store.