Audio glitches caused by memory pages being swapped out

Hypothesis

In case of high memory pressure, memory pages are getting swapped out. When pages used by the real-time audio thread are swapped out, it causes a delay. The real-time thread misses its deadline, and an audio glitch is introduced.

Detecting glitches

When the audio callback gets a new buffer, it is accompanied by a timestamp. From one call to the next, the timestamp increases by an amount equal to the buffer size. By recording the previous timestamp, it is trivial do detect missing buffers.

Statistical data

I develop and sell an audio editor, TwistedWave. It can detect glitches. When a glitch is detected, it places a marker, notifies the user, and inserts the right amount of silence to keep in sync with the audio.

TwistedWave will also send me a report when glitches occur, along with system information such as CPU and memory activity. From these reports, I can see that one recording session out of about 400 will get at least one glitch. The recording sessions are about one minute in length on average. I can also see that there is a strong correlation between the glitch rate and the CPU or memory usage. This is consistent with the hypothesis that the glitches are caused by memory pages being swapped out.

I have started investigating this in order to make TwistedWave even more reliable. Some users don’t experience glitches. Some experience a lot more glitches.

Locking pages in memory

With the mlock() call, it is possible to lock memory pages and prevent them from being swapped out. In order to avoid glitches, TwistedWave does its best to lock all the memory pages that contain the data structures used in the real-time thread, as well as the stack of the real-time thread, even though that should probably be the responsibility of CoreAudio.

I am not able to mlock the text pages that contain code used in the real-time thread. The mlock() call returns a permission denied error.

CoreAudio also does some mlocking. I can verify that by placing a breakpoint in mlock(), and starting a test program that will record audio. I can see that CoreAudio is doing a single mlock() call, apparently for a single data structure:

* thread #7, name = 'com.apple.audio.IOThread.client', stop reason = breakpoint 1.1
  * frame #0: 0x000000018a5a3708 libsystem_kernel.dylib`mlock
    frame #1: 0x000000018cf4cd10 CoreAudio`HALB_MlockFailHandling::_mlock(void const*, unsigned long) + 172
    frame #2: 0x000000018ccc77ec CoreAudio`HALB_SharedBuffer::Lock() + 184
    frame #3: 0x000000018cdfbcb4 CoreAudio`HALC_ProxyIOContext::IOWorkLoop() + 2452
    frame #4: 0x000000018cdfabf0 CoreAudio`invocation function for block in HALC_ProxyIOContext::HALC_ProxyIOContext(unsigned int, unsigned int) + 108
    frame #5: 0x000000018cf79f2c CoreAudio`HALC_IOThread::Entry(void*) + 88
    frame #6: 0x000000018a5da034 libsystem_pthread.dylib`_pthread_start + 136

I believe more mlock() calls are required in order to ensure glitch-free audio recording.

A call to mlockall() would help demonstrate that the audio glitches are caused by CoreAudio not locking enough memory, but unfortunately there is no such function on macOS.

Real-time thread stack being swapped out

I have built a test program that demonstrates that the real-time audio thread’s stack does get swapped out.

You can run that experiment by starting the attached RTThreadSwapped program. It will start the real-time audio thread, and the callback does nothing other than storing the address of the stack.

Meanwhile, the main thread repeatedly checks if the real-time audio thread stack is in memory or not, and prints a message when it gets swapped out.

While RTThreadSwapped is running, start a loop with the MemLoad program, and you will eventually get a message that says the stack was swapped out.

This does not happen often, but it does happen, and seeing it at least once is enough to prove the point.

Memory pressure causing glitches

Another experiment will show that a high memory pressure will introduce glitches.

Start the attached CountGlitches program. It will start recording, and do nothing else than counting glitches that are caused by missed buffers.

Start a loop with the MemLoad program, and see glitches coming in.

I have tested with a M2 macBook Pro, and glitches are coming less frequently than one per minute.

After running this test, I ran a sysdiagnose, and you can find it attached to this feedback. Before running sysdiagnose, two glitches occurred at these timestamps:

2024-02-13 16:26:25.258015 UTC
2024-02-13 16:28:06.251500 UTC

Included programs

Build them with the provided Makefile.

MemLoad

This program will allocate and mlock as much memory as possible.

It does a first pass, mlocking as much memory as possible until the mlock calls fail.

Then it continues allocating more memory, and stops when it has allocated as much memory as was mlocked in the first step.

In order to continuously load the memory, I run it in a loop like so:

while true; do ./MemLoad; done

CountGlitches

This program will start recording audio with the default input device, and report glitches on stdout.

It uses a small buffer size of 16 in order to make glitches more frequent.

The audio callback checks the timestamp to detect glitches, and increments a counter when one is found. It does nothing more, does not lock or allocate memory. All the memory used by the glitch counter has been mlocked().

When MemLoad is running to increase the memory pressure, glitches eventually get reported.

RTThreadSwapped

This program will demonstrate that the stack of the real-time audio thread can be swapped out of memory.

It will start recording audio, and do nothing in the audio callback other than storing the address of the stack. Meanwhile, the main thread will continuously call mincore() to check if the audio thread stack is still in memory.

When MemLoad is running to increase the memory pressure, the audio thread stack eventually gets swapped out.