How to symbolicate libart.so or libc.so stacktraces in Crashlytics Android NDK?

Question

Note: Symbols are showing up in crashlytics for our c++ library, the problem is that they aren't showing for system libraries like libc, libart, libbase, and libandroid_runtime.

We have some tricky crashes that happen entirely in the Android runtime and these are hard to debug without symbols. In firebase crashlytics we see the following stack trace:

Crashed: Thread :  SIGABRT  0x0000000000000000
#00 pc 0x4e574 libc.so 
#01 pc 0x4e540 libc.so 
#02 pc 0x5677d8 libart.so 
#03 pc 0x13ab0 libbase.so 
#04 pc 0x13090 libbase.so 
#05 pc 0x38cb6c libart.so 
#06 pc 0x39f7d8 libart.so 
#07 pc 0x1260e0 libandroid_runtime.so 
#08 pc 0x124ef4 libandroid_runtime.so 
#09 pc 0x124dc4 libandroid_runtime.so 
#10 pc 0x115468 libandroid_runtime.so 

When I force a test crash in our C++ library by dereferencing a null pointer, I see the following backtrace in my local Android Studio console:

...snip...
      #06 pc 00000000002d7644  /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+148) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #07 pc 00000000002cdd64  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #08 pc 00000000002f23d0  /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+312) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #09 pc 00000000003839f4  /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+800) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #10 pc 00000000003813f4  /apex/com.android.art/lib64/libart.so (MterpInvokeVirtualRange+1368) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #11 pc 00000000002c8714  /apex/com.android.art/lib64/libart.so (mterp_op_invoke_virtual_range+20) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
...snip...

However, the same crash in crashlytics looks like this:

...snip...
#07 pc 0x222244 libart.so 
#08 pc 0x218964 libart.so 
#09 pc 0x284208 libart.so 
#10 pc 0x3e34ac libart.so 
#11 pc 0x800ba4 libart.so 
...snip...

How can we get crashlytics to include the information that is clearly in the crashdump?

Some notes on our build setup:

• We have already followed https://firebase.google.com/docs/crashlytics/ndk-reports
• Gradle Project 1 builds the native library, ensuring that debug is on and symbols are not stripped.
• Gradle Project 2 builds the app and links in the library and tells crashlytics to upload native symbols

            firebaseCrashlytics {
                nativeSymbolUploadEnabled true
                unstrippedNativeLibsDir file("PATH/TO/UNSTRIPPED/DIRECTORY")
            }

Answer 1

This behavior is expected. When Crashlytics gets NDK crashes, these need to be symbolicated. In order to do this, the respective symbol files should be uploaded to Crashlytics.

With the configuration you mentioned, the symbols available in your app will be uploaded to Crashlytics.

nativeSymbolUploadEnabled true

But in the case of system libraries, the respective symbol files are not available (they are not public as far as I know). So Crashlytics won't have access to the required symbol files for symbolicating frames from system libraries.