Experimental setup
- Apps. The 100-app benchmark: 25 Popular, 25 Large, 25 Medium, and 25 Small real-world Google Play apps.
- Instrumentation. Every app is instrumented with AndroLog, which logs the classes, methods, statements, and components the app executes. No source code is needed.
- Environment. Tools run on an Android 11 (API level 30) Google APIs x86_64 emulator, with a dedicated virtual device per tool that is wiped at start-up.
- Budget. Each tool tests each app once, for 60 minutes. Coverage and crashes are read from the logcat output of the run.
- Human baseline. The same apps are explored manually by a human tester. This baseline is shown for reference and is not ranked.
Metrics
Code coverage
- Activities
- Share of the app's activities that were launched.
- Services
- Share of the app's services that were started.
- Receivers
- Share of the app's broadcast receivers that were triggered.
- Providers
- Share of the app's content providers that were accessed. No tool covered any provider, so this metric is shown but not ranked.
- Classes
- Share of the app's classes that were reached.
- Methods
- Share of methods that were executed.
- Statements
- Share of statements that were executed.
Every coverage metric is reported in two scopes. With libraries counts all code in the APK, third-party libraries included. App code only excludes library code, so it reflects how much of the app's own logic was exercised.
Crashes
RankDroid counts the unique crashes of the app under test in each run. Three kinds are detected in logcat:
- Java crashes (
FATAL EXCEPTION), identified by the root-cause exception and the top frames of the app's stack trace - ANRs (Application Not Responding), identified by the process, the component, and the kind of reason
- Native crashes (tombstones), identified by the signal, the normalised abort message, and the top native frames
Crashes with the same fingerprint are counted once per run. A crash counts only if its process belongs to the app under test.
From runs to rankings
- Final score. A tool's coverage score in a category is the mean of its end-of-run coverage over the apps of that category: 100 apps for Overall and 25 for every other category. Its crash score is the total number of unique crashes over those apps. An app without a run counts as 0.
- Evolution. Cumulative coverage and crashes are recorded every minute. Each run is aligned to its first observation, carried forward to 60 minutes, and starts at 0 at minute 0. Coverage curves are then averaged over the apps and crash curves are summed, in the same way as the final score. Because the curves stop at minute 60, their last point can be slightly below the final score of runs that lasted a little longer.
- All tools combined. For each app, the logs of all tools are merged, and coverage and crashes are computed on the merged log. This gives an empirical upper bound of what the tools reach together, shown as a red reference line.
- Ranking. Tools are ordered by their score, highest first. Coverage and crash counts are both "higher is better". Tied scores share the best rank, so a tie for second place is followed by fourth.
Reproducibility
All numbers on this site are generated from the raw run logs by scripts/build_results.py, which reproduces the analysis notebooks of the study. The data behind the views can be downloaded: the evolution page exports each curve as CSV next to the PDF figure, every tool page exports its per-app results, and the benchmark page exports the list of apps.
Keep in mind
- Each tool–app pair is run once, so the per-app results of randomised tools include run-to-run variance. Aggregating over 25 or 100 apps smooths this out.
- Results reflect each tool's publicly available implementation, run in the environment above. They may differ from the numbers reported in the tool's own paper.
- Coverage is measured on instrumented APKs. If no coverage is recorded for an app, for example because a tool cannot run on it, that app counts as 0 in the tool's mean.