Constant Graphics Crashes on most mobile devices after 2021

Zirokden June 26, 2023, 8:04am 1

We did an update from current 2021 LTS to current 2022 LTS (2022.2.1f1)

We are experiencing high amount of crashes, apparently graphics related.

One interesting thing we noticed is that on crash logs there are usually 40+ threads open - we are uncertain what it could be.

Iphone:

Crash 1:

Unknown Function (/usr/lib/libobjc.A.dylib) Thread 31 (crashed) 0 libobjc.A.dylib 0x00000001d2081178 <symbols missing for uuid: f6368be9109b3405b282624040529afd> 1 UnityFramework 0x0000000000aee61c Reset (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/metal/BuffersMetal.h:0) 2 UnityFramework 0x000000010c24261c EndWrite (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/metal/BuffersMetal.mm:345) 3 UnityFramework 0x000000010c250ad0 EndBufferWrite (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/metal/GfxDeviceMetal.mm:3434) 4 UnityFramework 0x0000000000916f4c PutGeometryJobTaskInternal (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/GeometryJob.cpp:180)

Crash 2:

Thread 31 (crashed) 0 UnityFramework 0x000000010b0bffa8 GetVersion (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/GfxVersionList.cpp:106) 1 UnityFramework 0x000000010b2add2c EndWrite (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/metal/BuffersMetal.h:75) 2 UnityFramework 0x000000010b2adcd0 EndBufferWrite (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/metal/GfxDeviceMetal.mm:3434) 3 UnityFramework 0x000000000091414c PutGeometryJobTaskInternal (/Users/bokken/build/output/unity/unity/Runtime/GfxDevice/GeometryJob.cpp:180)

Android:

Thread 0 (crashed) 0 libunity.so 0x000000767d376480 BufferGLES::EndWrite(unsigned long) 1 libunity.so 0x000000767d3763c8 GfxDeviceGLES::EndBufferWrite(GfxBuffer*, unsigned long) 2 libunity.so 0x000000767d32cfa4 GeometryJobTasks::PutGeometryJobTask(GfxDevice&, GeometryJobTasks::GeometryJobTask*) 3 libunity.so 0x000000767d32ce90 GeometryJobTasks::EndFrame(GfxDevice&)

Have a feeling they are related somehow but not sure what exactly could be causing this.

We have tried disabling multi-thread rendering for instance and it still happens.

Usually when it crashes we see 50+ threads open in the following state:

Thread 2 0 libsystem_kernel.dylib 0x00000001f0e014dc <symbols missing for uuid: 9ab02ad0348c30cb99f40979c2d47515> 1 libdispatch.dylib 0x00000001b8fcd674 <symbols missing for uuid: e3ea4f635d11342aaf199f58dbc8e259> 2 UnityFramework 0x0000000000c86814 Acquire (/Users/bokken/build/output/unity/unity/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:37) 3 UnityFramework 0x0000000000c86808 Baselib_SystemSemaphore_Acquire (/Users/bokken/build/output/unity/unity/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:62) 4 UnityFramework 0x000000010d3da814 Baselib_SystemSemaphore_Acquire (/Users/bokken/build/output/unity/unity/External/baselib/builds/Source/CProxy/Baselib_SystemSemaphore_CProxy.inl.h:19) 5 UnityFramework 0x000000000033b710 Baselib_Semaphore_Acquire (/Users/bokken/build/output/unity/unity/External/baselib/builds/Include/C/Internal/Baselib_Semaphore_SemaphoreBased.inl.h:0) 6 UnityFramework 0x000000000033b708 WaitForSignalNoProfile (/Users/bokken/build/output/unity/unity/Runtime/Threads/Semaphore.h:33) 7 UnityFramework 0x000000010ca8f710 Run (/Users/bokken/build/output/unity/unity/Runtime/Misc/GarbageCollectSharedAssets.cpp:272) 8 UnityFramework 0x000000010ca93450 RunThread (/Users/bokken/build/output/unity/unity/Runtime/Misc/GarbageCollectSharedAssets.cpp:250) 9 UnityFramework 0x000000010cb5f51c RunThreadWrapper (/Users/bokken/build/output/unity/unity/Runtime/Threads/Thread.cpp:107) 10 libsystem_pthread.dylib 0x000000022a4f99ac <symbols missing for uuid: a8989ffb141439dca1a81534f21923fe> 11 libsystem_pthread.dylib 0x000000022a4f8e68 <symbols missing for uuid: a8989ffb141439dca1a81534f21923fe>

We turned OFF incremental GC to see if could be causing the issue and apparently is not.

Regards, Gabriel

troylightfox June 26, 2023, 10:24pm 2

We’ve been seeing similar crashes across all renderer types(Vulkan, Metal, DX11). Unity support just mentioned this is due to be fixed in 2022.3.4f1, released on 6/28. Probably back ported to other LTS versions. Waiting to hear back on root cause.

jperryoddgames June 29, 2023, 2:58am 3

We’ve also got a lot of these crashes coming through. Have recently upgraded from unity 2021.3.xx to unity 2022.3.0f1.
Trace looks pretty much the same as what’s been posted above but I’ll also include ours below.

#00 pc 0x0000000000b35f94 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (BufferGLES::EndWrite(unsigned long)) #01 pc 0x0000000000b35edc /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GfxDeviceGLES::EndBufferWrite(GfxBuffer*, unsigned long)) #02 pc 0x0000000000afcf08 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GeometryJobTasks::PutGeometryJobTask(GfxDevice&, GeometryJobTasks::GeometryJobTask*)) #03 pc 0x0000000000afcdf4 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GeometryJobTasks::EndFrame(GfxDevice&)) #04 pc 0x0000000000cf7cc4 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)) #05 pc 0x0000000000cf0ffc /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)) #06 pc 0x0000000000cf0d44 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (GfxDeviceWorker::RunGfxDeviceWorker(void*)) #07 pc 0x0000000000664bc0 /data/app/~~auvFIy1Ag2jlXFIwIENnVg==/com.project-oPD3vSAOEHlJqg8q7uHFog==/lib/arm64/libunity.so (Thread::RunThreadWrapper(void*)) #08 pc 0x00000000000efb14 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+264) #09 pc 0x000000000008c35c /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

AcidArrow June 29, 2023, 4:23am 4

Try disabling multithreaded rendering.

jperryoddgames June 29, 2023, 5:46am 5

It looks like Unity 2022.3.4f1 has now released and is available for download in Unity Hub. Cannot see it linked in the download archive online or find release notes yet so I’m unable to confirm if it mentions this particular issue.

@troylightfox mentioned above that this issue is apparently scheduled to be fixed with this new version.
We’re going to try a small release tonight with this new version to see if things improve; otherwise we’ll try disabling multi-threaded rendering.

Zirokden October 20, 2023, 12:25pm 6

Disabling multithread rendering fixes the issue but makes the performance garbage.

troylightfox October 23, 2023, 5:21pm 7

I can report that for us, upgrading to 2022.3.4f1 fixed this issue. We’ve since upgraded to 2022.3.7f1 and continue to not experience this crash in production. Particularly it was this: