In case of high memory pressure, memory pages are getting swapped out. When pages used by the real-time audio thread are swapped out, it causes a delay. The real-time thread misses its deadline, and an audio glitch is introduced.
When the audio callback gets a new buffer, it is accompanied by a timestamp. From one call to the next, the timestamp increases by an amount equal to the buffer size. By recording the previous timestamp, it is trivial do detect missing buffers.
I develop and sell an audio editor, TwistedWave. It can detect glitches. When a glitch is detected, it places a marker, notifies the user, and inserts the right amount of silence to keep in sync with the audio.
TwistedWave will also send me a report when glitches occur, along with system information such as CPU and memory activity. From these reports, I can see that one recording session out of about 400 will get at least one glitch. The recording sessions are about one minute in length on average. I can also see that there is a strong correlation between the glitch rate and the CPU or memory usage. This is consistent with the hypothesis that the glitches are caused by memory pages being swapped out.
I have started investigating this in order to make TwistedWave even more reliable. Some users don’t experience glitches. Some experience a lot more glitches.
With the mlock()
call, it is possible to lock memory
pages and prevent them from being swapped out. In order to avoid
glitches, TwistedWave does its best to lock all the memory pages that
contain the data structures used in the real-time thread, as well as the
stack of the real-time thread, even though that should probably be the
responsibility of CoreAudio.
I am not able to mlock the text pages that contain code used in the
real-time thread. The mlock()
call returns a permission
denied error.
CoreAudio also does some mlocking. I can verify that by placing a
breakpoint in mlock()
, and starting a test program that
will record audio. I can see that CoreAudio is doing a single
mlock()
call, apparently for a single data structure:
* thread #7, name = 'com.apple.audio.IOThread.client', stop reason = breakpoint 1.1
* frame #0: 0x000000018a5a3708 libsystem_kernel.dylib`mlock
frame #1: 0x000000018cf4cd10 CoreAudio`HALB_MlockFailHandling::_mlock(void const*, unsigned long) + 172
frame #2: 0x000000018ccc77ec CoreAudio`HALB_SharedBuffer::Lock() + 184
frame #3: 0x000000018cdfbcb4 CoreAudio`HALC_ProxyIOContext::IOWorkLoop() + 2452
frame #4: 0x000000018cdfabf0 CoreAudio`invocation function for block in HALC_ProxyIOContext::HALC_ProxyIOContext(unsigned int, unsigned int) + 108
frame #5: 0x000000018cf79f2c CoreAudio`HALC_IOThread::Entry(void*) + 88
frame #6: 0x000000018a5da034 libsystem_pthread.dylib`_pthread_start + 136
I believe more mlock()
calls are required in order to
ensure glitch-free audio recording.
A call to mlockall()
would help demonstrate that the
audio glitches are caused by CoreAudio not locking enough memory, but
unfortunately there is no such function on macOS.
I have built a test program that demonstrates that the real-time audio thread’s stack does get swapped out.
You can run that experiment by starting the attached RTThreadSwapped program. It will start the real-time audio thread, and the callback does nothing other than storing the address of the stack.
Meanwhile, the main thread repeatedly checks if the real-time audio thread stack is in memory or not, and prints a message when it gets swapped out.
While RTThreadSwapped is running, start a loop with the MemLoad program, and you will eventually get a message that says the stack was swapped out.
This does not happen often, but it does happen, and seeing it at least once is enough to prove the point.
Another experiment will show that a high memory pressure will introduce glitches.
Start the attached CountGlitches program. It will start recording, and do nothing else than counting glitches that are caused by missed buffers.
Start a loop with the MemLoad program, and see glitches coming in.
I have tested with a M2 macBook Pro, and glitches are coming less frequently than one per minute.
After running this test, I ran a sysdiagnose, and you can find it attached to this feedback. Before running sysdiagnose, two glitches occurred at these timestamps:
2024-02-13 16:26:25.258015 UTC
2024-02-13 16:28:06.251500 UTC
Build them with the provided Makefile.
This program will allocate and mlock as much memory as possible.
It does a first pass, mlocking as much memory as possible until the mlock calls fail.
Then it continues allocating more memory, and stops when it has allocated as much memory as was mlocked in the first step.
In order to continuously load the memory, I run it in a loop like so:
while true; do ./MemLoad; done
This program will start recording audio with the default input device, and report glitches on stdout.
It uses a small buffer size of 16 in order to make glitches more frequent.
The audio callback checks the timestamp to detect glitches, and increments a counter when one is found. It does nothing more, does not lock or allocate memory. All the memory used by the glitch counter has been mlocked().
When MemLoad is running to increase the memory pressure, glitches eventually get reported.
This program will demonstrate that the stack of the real-time audio thread can be swapped out of memory.
It will start recording audio, and do nothing in the audio callback
other than storing the address of the stack. Meanwhile, the main thread
will continuously call mincore()
to check if the audio
thread stack is still in memory.
When MemLoad is running to increase the memory pressure, the audio thread stack eventually gets swapped out.