Uncovering the Secret Tools in Google’s Oreo NDK
Using the hidden tools!
8/21/2017 wasn’t only a day for celestial bodies to be aligned. It was also the day that Google announced their latest Operating System to grace the mobile devices of the world. Google’s use of the eclipse to create extra fanfare around Oreo was not lost on the media.
Every few releases, something notable is changed that has direct positive impact on developers. Oreo was one such release. With Oreo a subtle change left nearly undocumented, yet drastically affects how Native Mobile developers should approach creating professional change management pipelines.
Memory sanitization, unit and integration testing, and profiling were the realm of the most adventurous developer. Oreo’s release delivers a shift where any native developer can now utilize a classical professional tooling environment.
Most notably, VR development is in constant pursuit of getting everything done in constant, smooth, 60 Frames Per Second (FPS). Anything slower than that frame rate is a trip down the vomit comet app trail of bad reviews. This is why game developers are so at home in VR development as classical game development is a high FPS optimization love affair.
Under the desire of faster FPS, certain techniques have been developed over the years that, while highly efficient, yield complexity. Complexity is the path of error-prone code.
Threading, memory management, networking, audio, graphics, and logic in general at this frame rate provides fertile ground for cryptic bugs that always seem to look right. The more complex an app gets, with the more team members, the more bugs creep in that are usually among the more difficult to solve.
With VR development’s highly complex code requirements, getting help from an automated tool is vital towards wrangling the bugs. In Oreo, if an APK is packaged with wrap.sh, the OS will execute that wrap.sh script when launching the app. That gives developers a chance to use their normal classical tools to launch the app and put it under a watchful eye.
The tools that all developers know and love, from memory sanitizing tools (ASan, Valgrind), thread sanitizing tools (TSan), unit testing tools (gtest, mockito), and even profiling tools (prof) can now stand at the ready in your developer’s toolbox to help diagnose problems at runtime.
Each of these robust, mature tools are normally used in other platforms. They were designed years ago, with the idea of launching an application, capturing that applications’ output, memory allocation, or system performance. It is also the model that debuggers frequently use; however, for wise security reasons, Android has restricted this behavior from end user devices.
The workflow of yesterday to utilize these tools, involved first creating a setup of each project with Valgrind rebuilt for each ABI under test. Then running the app via the adb command line disjoint from the Android Studio debugger. Finally, create some custom scripting which involves a dance of shipping C symbols, .oat files and android os root level access magic, all in order to achieve a simple app wrapping start script.
Getting Valgrind to work on Android was always one of the more time consuming tasks with getting any project configured. In this article, I’m going to detail how to use ASan, a tool that works similarly to Valgrind and helpfully is shipped with the NDK.
Prior to Oreo’s wrap.sh, the method for getting an app startup script involved being able to write to the /system partition. Unfortunately, that is a major security vulnerability so commercial devices strongly lock that access down, as well they should.
However, as a legitimate use case for having that access, I will break with what I normally tell users. The first step, was root the device, then unlock the bootloader, then install a eng/dev version of the OS (which usually doesn’t exist), and finally unlock the ability to write to /system. All of that prior to getting an ASan startup script and binary installed.
Google recognized this was a problem a few years back and started shipping the emulator with a eng/dev version of the OS. This is probably the best recommended approach for running bugs. Load up the app in a startup script running on the emulator which allows for eng/dev system level write access.
Now VR development isn’t normal development. We have our own problems that the emulator can’t solve. Oculus Mobile is a Samsung only thing and the SDK checks for a Samsung OS, thus it won’t even run on the emulator. In addition, the emulator doesn’t have an ability to “look around” which is a core requirement for VR. Thus we’re a device only requirement for most of our tooling.
How the world of our tooling changed when Google released Oreo’s ability to have wrap.sh! Praise be and pass the sanitizers! Unfortunately, despite this great news, not all is completely easy for VR developers even with Google’s latest Android incantation.
Google shipped with a bug in the first version of Oreo surrounding wrap.sh. This bug required us to have root capabilities in order to turn off selinux (temporarily). This bug has been addressed! The most recent version of Oreo 8.1 includes support without rooting! Ensure you have version 17 of the NDK (very latest) and Android Studio. Everything can be done from Android Studio and your individual app.
The rest of the way, I’ll demo how to use ASan in any project. First your app must be compiled with “-fsanitize=address -fno-omit-frame-pointer” flags. Next it must be linked with “-fsanitize=address” link flags. NB: these flags will cause the app to NOT work without ASan, so making a build flavor is highly recommended (as is only doing this for debug build types). As the app will only run if it can find an ASan runtime, we must package the APK with the ASan runtime library and use LD_PRELOAD to instruct the OS where to find the ASan runtime library.
Concretely, let’s put all that in some gradle code:
1.) In the main project level gradle, let’s find the path to the NDK set it to a globally accessible variable, and a global boolean flag the developer can set to turn ASan on/off. The following gradle will achieve that:
Properties properties = new Properties()
properties.load(project.rootProject.file('local.properties').newDataInputStream())
project.ext {
useASAN = true
ndkDir = properties.getProperty('ndk.dir')
}2.) Now in the project’s app module gradle; where we communicate with CMake. It’s probably a good idea to inform CMake that this is an ASan build so it can setup the required build flags. (if you can’t edit CMakeLists.txt, I suggest liberal use of the externalNativeBuild’s arguments DSL).
buildTypes {
debug {
externalNativeBuild {
cmake {
if (rootProject.ext.useASAN)
arguments "-DUSEASAN=ON"
}
}
}
}3.) Now in CMakeLists.txt, which instructs the build system how to build the native libraries, do the following to set our compile and link flags appropriately.
if(USEASAN)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address -fno-omit-frame-pointer")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=address -fno-omit-frame-pointer")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fsanitize=address")
set(CMAKE_STATIC_LINKER_FLAGS "${CMAKE_STATIC_LINKER_FLAGS} -fsanitize=address")
endif(USEASAN)NB: one could use target_* family of CMake directives, however, I chose to demo setting it project wide for convenience in case the developer uses other modules.
4.) Next we need to create the shell script that Oreo will use to launch our app with ASan running:
static def writeWrapScriptToFullyCompileJavaApp(wrapFile, abi) {
if(abi == "armeabi" || abi == "armeabi-v7a")
abi = "arm"
if(abi == "arm64-v8a")
abi = "aarch64"
wrapFile.withWriter { writer ->
writer.write('#!/system/bin/sh\n')
writer.write('HERE="$(cd "$(dirname "$0")" && pwd)"\n')
writer.write('export ASAN_OPTIONS= allow_user_segv_handler=1\n')
writer.write('export ASAN_ACTIVATION_OPTIONS=include_if_exists=/data/local/tmp/asan.options.b\n')
writer.write("export LD_PRELOAD=\$HERE/libclang_rt.asan-${abi}-android.so\n")
writer.write('\$@\n')
}
}In my writeWrapScript gradle function, I have set ASan to look for the file asan.options.b for any user defined options. This is predominately a convenience as changing the ASan options would otherwise require rebuilding the whole app. I find this helpful when dealing with headsets being physically not near my development machine. With this approach, I can edit the file with any local app (or CI environment/tester can set those dynamically), relaunch it and get different results; all without plugging into my development machine; making finding memory leaks a tester’s capability.
5.) Next I need to call that wrap script gradle function during a build, place it in a directory that is useful, and copy the asan runtime libraries to a useful directory. All of that can be done with these two gradle tasks:
task copyASANLibs(type:Copy) {
def libDir = file("$rootProject.ext.ndkDir").absolutePath + "/toolchains/llvm/prebuilt/"
for (String abi : SupportedABIs) {
def dir = new File("app/wrap_add_dir/libs/" + abi)
dir.mkdirs()
if(abi == 'armeabi-v7a' || abi == 'armeabi')
abi = "arm"
if(abi == "arm64-v8a")
abi = "aarch64"
FileTree tree = fileTree(dir: libDir).include("**/*asan*${abi}*.so")
tree.each { File file ->
from file
into dir.absolutePath
}
}
}
task createWrapScriptAddDir(dependsOn: copyASANLibs) {
for (String abi : SupportedABIs) {
def dir = new File("app/wrap_add_dir/res/lib/" + abi)
dir.mkdirs()
def wrapFile = new File(dir, "wrap.sh")
writeWrapScriptToFullyCompileJavaApp(wrapFile, abi)
println "write file " + wrapFile.path
}
}6.) Now to automatically attach our tasks to the gradle build process and clean up after ourselves when a clean command is issued:
tasks.whenTaskAdded { task ->
if (task.name.startsWith('generate')) {
if(rootProject.ext.useASAN)
task.dependsOn createWrapScriptAddDir
}
}
task deleteASAN(type: Delete) {
delete 'wrap_add_dir'
}clean.dependsOn(deleteASAN)
7.) Final step is to tell the Android gradle plugin where to find our wrap.sh file and libraries:
if (rootProject.ext.useASAN) {
sourceSets {
main {
jniLibs {
srcDir {
"wrap_add_dir/libs"
}
}
resources {
srcDir {
"wrap_add_dir/res"
}
}
}
}
}Now we can bask in the awesomeness that is ASan. Even better, as ASan will throw a sigabt anytime there’s a memory violation, Android Studio’s LLVM integration works with it natively:
To see an example project that has everything detailed in a real world working project template: https://github.com/gpx1000/ExampleASAN
Did you like this post? Please check out my blog at www.gpxblog.com for more mobile developer posts; and please stay tuned to VRPop for more VR/AR specific developer posts.
Update: March 28 2018 Prior to Oreo 8.1, there was a need to turn SELinux off. This post was written prior to 8.1 being available. For the sake a legacy, I’m leaving the below instructions intact incase someone has need of them. If you are on 8.1, stop here, avoid rooting your device or turning off security features.
Finally, If and only if you had to root your device or. Due to the Google bug in Oreo, we need to disable SElinux prior to running with ASan. This only needs to happen once per OS session (i.e. do it every time the device starts). As a practice, for security reasons, I recommend turning off SElinux whenever finishing a testing session and turning it on whenever starting a testing session. Each testing session can have many multiple app runs; but the general use case is you’re knowingly making the device less secure to do your testing; good practice dictates turning security back on when done.
The following two gradle tasks will accomplish this:
task setEnforceOff() {
doLast {
def sdkDir = file("$rootProject.ext.ndkDir").getParentFile().absolutePath
def process = ("$sdkDir/platform-tools/adb shell su -c \"setenforce 0\"").execute()
process.waitFor()
if (process.exitValue()) {
println("stdout: " + process.in.text)
println("stderr: " + process.err.text)
throw new GradleException("error in set enforce command")
}
}
}
task setEnforceOn() {
doLast {
def sdkDir = file("$rootProject.ext.ndkDir").getParentFile().absolutePath
def process = ("$sdkDir/platform-tools/adb shell su -c \"setenforce 1\"").execute()
process.waitFor()
if (process.exitValue() != 0) {
println("stdout: " + process.in.text)
println("stderr: " + process.err.text)
throw new GradleException("error in set enforce command")
}
}
}Or just run “adb shell su -c setenforce 0” to turn it off and “adb shell su -c setenforce 1” to turn it back on from the command line of any computer attached to it via adb.
