[old thread] Automatic mixing with a home made computer program

[des0free]

In my work I sometimes process experimental "signal vs. time" data with custom written computer codes.

So I thought it would be fun to see if I could write code to "automatically roughly mix" audio tracks. It reads multiple audio tracks (WAV files), analyzes and auto-adjust the levels of each track/instrument to be *roughly* correct in the mix, applies basic lowpass and highpass EQ filters to each track, mixes the tracks (by simple summation), and outputs a mixed track adjusted to have maximum volume without clipping.  My idea is this produces a rough mix automatically and could output pre-processed tracks to be imported into a DAW for further tweaking.

Listen to the attached audio output (from my cover of "massachusetts"; 8 tracks: drums, bass, 2 keys, 2 guitar, 2 vocal).  The only adjustment I made manually was to multiply the bass signal by 0.4 since it initially sounded too loud.  I used Reaper (DAW) only to apply a compressor and limiter to the mix and convert to MP3. The result is not perfect, but not bad for a novice and is a starting point for further development!

UPDATES:
- Now has STEREO processing with 32-bit WAV data and high/low pass filtering
- See posts below for hybrid approach where compression and reverb are added to each track in a DAW, then my code auto mixes:
https://songcrafters.org/forum/index.php?topic=31414.msg373562#msg373562
https://songcrafters.org/forum/index.php?topic=31414.msg373563#msg373563

Matlab code:

[ydrumSt,Fs] = audioread('DrumSt.wav','native');
[ybassSt,Fs] = audioread('BassSt.wav','native');
[yguitLSt,Fs] = audioread('GuitLSt.wav','native');
[yguitRSt,Fs] = audioread('GuitRSt.wav','native');
[ykeySt,Fs] = audioread('Key1St.wav','native');
[ykey2St,Fs] = audioread('Key2St.wav','native');
[yvocSt,Fs] = audioread('VocSt.wav','native');
[yvoc2St,Fs] = audioread('Voc2St.wav','native');

ydrum=double(ydrumSt);
ybass=double(ybassSt);
yguitL=double(yguitLSt);
yguitR=double(yguitRSt);
ykey=double(ykeySt);
ykey2=double(ykey2St);
yvoc=double(yvocSt);
yvoc2=double(yvoc2St);

% high pass filter
ydrum = highpass(ydrum,31,Fs);
ybass = highpass(ybass,40,Fs);
yguitL = highpass(yguitL,150,Fs);
yguitR = highpass(yguitR,150,Fs);
ykey = highpass(ykey,70,Fs);
ykey2 = highpass(ykey2,70,Fs);
yvoc = highpass(yvoc,100,Fs);
yvoc2 = highpass(yvoc2,100,Fs);

% normalize each by max
maxdrum=max(abs(ydrum(:,1))); ydrum=(2^31/maxdrum)*ydrum;
maxbass=max(abs(ybass(:,1))); ybass=(2^31/maxbass)*ybass;
maxguitL=max(abs(yguitL(:,1))); yguitL=(2^31/maxguitL)*yguitL;
maxguitR=max(abs(yguitR(:,2))); yguitR=(2^31/maxguitR)*yguitR;
maxkey=max(abs(ykey(:,1))); ykey=(2^31/maxkey)*ykey;
maxkey2=max(abs(ykey2(:,2))); ykey2=(2^31/maxkey2)*ykey2;
maxvoc=max(abs(yvoc(:,1))); yvoc=(2^31/maxvoc)*yvoc;
maxvoc2=max(abs(yvoc2(:,2))); yvoc2=(2^31/maxvoc2)*yvoc2;

% compute RMS above threshold for "average volume" estimate
thresh=0.1e8;
ldrum=rms(ydrum(ydrum>thresh));
lbass=rms(ybass(ybass>thresh));
lguitL=rms(yguitL(yguitL>thresh));
lguitR=rms(yguitR(yguitR>thresh));
lkey=rms(ykey(ykey>thresh));
lkey2=rms(ykey2(ykey2>thresh));
lvoc=rms(yvoc(yvoc>thresh));
lvoc2=rms(yvoc2(yvoc2>thresh));

% Adjust and tweak levels
ydrum=1*2^28/ldrum*ydrum;
ybass=0.4*2^28/lbass*ybass;
yguitL=1*2^28/lguitL*yguitL;
yguitR=1*2^28/lguitR*yguitR;
ykey=1*2^28/lkey*ykey;
ykey2=1*2^28/lkey2*ykey2;
yvoc=1*2^28/lvoc*yvoc;
yvoc2=1*2^28/lvoc2*yvoc2;

% sum and low pass filter
ytot=ydrum+ybass+yguitL+yguitR+ykey+ykey2+yvoc+yvoc2;
ytot=lowpass(ytot,20000,Fs);

% Maximize output level
maxytot=max(ytot); maxytot=max(maxytot);
ytot(:,1)=(2^31/maxytot)*ytot(:,1);
ytot(:,2)=(2^31/maxytot)*ytot(:,2);

ytot=int32(ytot);
audiowrite('out2.wav',ytot,Fs,'BitsPerSample',32);

[Mach]

This sounds like a very interesting concept, but I know very little about code. I have worked with 'modifying' code in Native Instruments Kontakt to change or view certain parameters but that's very basic stuff. When you say "automatic mixing" are you trying to develop a way to build your own DAW software to have auto faders, knobs and fx?

I know there are different Mastering software's that can do EQ and compression in real time. I'm not sure if they can be programmed to be automated. So on the tracks you have here are they coded to be muted and/or soloed like on a DAW track strip?

Mach

[64Guitars]

In my work I process experimental data, often "signal vs. time" with custom written computer codes.

An audio track is similar so I thought it would be fun to see if I could make a very simple code to read multiple tracks (as WAV files), plot the signals, adjust level of each, sum them together, try to maximize overall volume without clipping, and output to wav.

Sounds like fun.

However...

My dream is to make a code that would automatically mix and master (and eventually incorporate EQ, compression, etc too)

Automatically? Mixing and mastering is as much art as it is science. So you can't automate it and expect to get good results because every recording is different and choices have to be made about relative track levels, panning, EQ, loop effects, etc. These are personal choices that can't be hard-coded into a program (at least, not without a lot of complex A.I.) because every song is different and every person will make different choices based on their own tastes and experience.

Perhaps a more useful project would be to write a program that will process a single track to create a cool effect. Eventually, after you've written several effects, you could combine them into a single program and let the user select which effect(s) to apply.

[des0free]

This sounds like a very interesting concept, but I know very little about code. I have worked with 'modifying' code in Native Instruments Kontakt to change or view certain parameters but that's very basic stuff. When you say "automatic mixing" are you trying to develop a way to build your own DAW software to have auto faders, knobs and fx?

I know there are different Mastering software's that can do EQ and compression in real time. I'm not sure if they can be programmed to be automated. So on the tracks you have here are they coded to be muted and/or soloed like on a DAW track strip?

Mach

My idea is to computationally analyze each track's (instrument's) signal and process it so that its level in the mix will be approximately correct, regardless of the level it was recorded, following my rough tastes for mixing different instruments (e.g., how much bass I usually like).  Also I will apply high pass and low pass filters with cutoff values I typically use (e.g., my vocal high pass filtered around 100 Hz, and everything low pass filtered at 20,000 Hz).  My thought is this will give me a quick starting point from which to make further tweaks depending on the nature of the song.  For example, my program could "pre-process" the tracks and they could then be imported into a DAW for further adjustments.

[des0free]

Mixing and mastering is as much art as it is science. So you can't automate it and expect to get good results because every recording is different and choices have to be made about relative track levels, panning, EQ, loop effects, etc. These are personal choices that can't be hard-coded into a program (at least, not without a lot of complex A.I.) because every song is different and every person will make different choices based on their own tastes and experience.

I agree, but I hope to computationally analyze each track's (instrument's) signal and process it so that its level in the mix will be approximately correct, regardless of the level it was recorded, following my rough tastes for mixing different instruments (e.g., how much bass I usually like).  Also I can auto-apply high pass and low pass filters with cutoff values I typically use.  This will give me a quick starting point from which to make further tweaks depending on the nature of the song.  For example, my program could "pre-process" the tracks before they are imported into a DAW for further adjustments.

[des0free]

UPDATE: I improved the quality and relative levels in the auto-mixed output track by using 32-bit WAV files, converted to double-precision floating point to process gain and apply lowpass and highpass filters and mix, then converted to 32-bit integer to write to output WAV file.  I then used a DAW (Reaper) only to apply a compressor and limiter and to convert to MP3 to post the audio - check out my new audio file in my original post.

[des0free]

Update: My code can now auto-mix in STEREO - original post updated with an example audio file.

[des0free]

Update - I implemented an option to "treble boost" above 4000 kHz on the vocal track (by high pass filtering a copy of the track then adding a fraction of it to the original), and a "bass cut" below 600 Hz on the keyboard tracks, plus added additional high and low pass filters.  And experimented with tweaking track levels via fiddling numbers in my code (rather than using a DAW).

Filters:
Drums: highpass @ 35 Hz
Bass: highpass @ 35 Hz, lowpass @ 3000 Hz
Keys: highpass @ 100 Hz
Acoustic Guitars: highpass @ 100 Hz, lowpass @ 9000 Hz
Vocals: highpass @ 70 Hz

Manual Level adjustments (beyond the levels calculated automatically in the code):
Drum signal multiplied by 1.2
Bass multiplied by 0.5
Guitars multiplied by 1.25
Key1 multiplied by 1.7
Key2 multiplied by 0.8
Vocals unchanged

So it shows that the automatic level adjustment is pretty good, because relatively small adjustments beyond it are needed (not that I'm claiming this is a great mix...because I don't consider myself to have great ears for mixing...).

[des0free]

I decided to try pre-processing each track (from "Massachusetts") in Reaper (DAW) using plugins to add compression (to guitars, bass, and vocals), bass amp sim, and reverb (to keys and vocals), without having to worry at all about relative levels of each track (keeping them low), then output each track to WAV and use my code to "auto mix".  After doing this my mixing algorithm seemed to work even better, probably because my level estimator works better on pre-compressed tracks.  The only adjustment I made by hand was to multiply one keyboard (strings) track by 2.  This yielded the attached audio.

Eventually I might be able to figure out how to add compression and reverb automatically in my code...

[des0free]

And here is the same approach described in my last post to a different song (my cover of "Have you ever seen the rain").  In this case I did a bit more minor manual tweaking after the auto-mix algorithm.

Bass multiplied by 1.5
Guitar multiplied by 2 (was not doubletracked)
Vocal multiplied by 3 (was not doubletracked)
Drums and Keys unchanged