Using Gradle to split external libraries in separated dex files to solve Android Dalvik 64k methods limit

Question

Is there a proper/easy way to solve the 64k methods limit using Gradle?

I mean some custom Gradle task to use pre-dexed jars to create separated dex fil

Answer 1

I am the maintainer of https://github.com/creativepsyco/secondary-dex-gradle/ and I am a gradle n00b, therefore I chose the path of BASH scripts although I think it can be done directly in the build file. OR can be refactored to run as a plugin, i might do that when I am upto to terms with Gradle. Here is the reason for my logic.

In order to understand how to split the DEX you must know the build system task order. If you are using gradle then you must know that there are a series of tasks injected inside the build cycle.

For example:

:sdk:processReleaseJavaRes UP-TO-DATE
:sdk:packageReleaseJar
:sdk:compileReleaseNdk UP-TO-DATE
:sdk:packageReleaseJniLibs UP-TO-DATE
:sdk:packageReleaseLocalJar UP-TO-DATE
:sdk:packageReleaseRenderscript UP-TO-DATE
:sdk:packageReleaseResources UP-TO-DATE
:sdk:bundleRelease
:app:prepareComAndroidSupportAppcompatV71910Library UP-TO-DATE
:app:prepareComFacebookAndroidFacebook3141Library UP-TO-DATE
:app:prepareDebugDependencies
:app:compileDebugAidl UP-TO-DATE
:app:compileDebugRenderscript UP-TO-DATE
:app:generateDebugBuildConfig UP-TO-DATE
:app:generateDebugAssets UP-TO-DATE
:app:mergeDebugAssets UP-TO-DATE
:app:generateDebugResValues UP-TO-DATE
:app:generateDebugResources UP-TO-DATE
:app:mergeDebugResources UP-TO-DATE
:app:processDebugManifest UP-TO-DATE
:app:processDebugResources UP-TO-DATE
:app:generateDebugSources UP-TO-DATE
:app:compileDebugJava
:app:preDexDebug
:app:dexDebug
:app:processDebugJavaRes UP-TO-DATE
:app:validateReleaseConfigSigning
:app:packageDebug
:app:zipalignDebug
:app:assembleDebug

In order to do Dexing you should be able to be able to inject your custom task between the dex* and the process* tasks. If you can do this, then Multiple DEXing becomes easy.

The Bash script here essentially does this, if you examine the debug task, it will basically:

• Get the Library Jar file to dex, usually it is build specific & exists in the exploded-aar folder for Android Libraries & run the DEX tool on it
• Copy this over to the assets folder, which exists in the final libs folder to be packaged inside the app
• All the library resources etc are already merged, which means there is a need to unzip & zip the file again.

In the gradle build script

 // For Debug simply remove the library from getting dex and create it
                //----------------------- Extra Debug Step ----------------//
                def libraryFiles = new ArrayList<?>()
                def secondaryFile = new ArrayList<?>()

                variant.dex.libraries.each {
                    File file ->
                        if (!file.absolutePath.contains("lib/unspecified/classes.jar")) {
                            libraryFiles.add(file)
                        } else {
                            secondaryFile.add(file)
                        }
                }
                variant.dex.libraries = libraryFiles
                //----------------------- Extra Debug Step ----------------//

                packagingTask.dependsOn variant.javaCompile
            }

This manually removes the library from getting dexed, so that it can be generated via the bash script.

I think you can figure out the dexing during the release process in the same way. The other important thing to note is that the Proguard Task is controlled by android gradle plugin and you can't change much about it. Issue with Proguard Rules:

• Each pass of proguard is different, we don't want to end up in a situation where our 2 DEXes have different proguard mappings
• This leaves us in a situation where we cannot proguard our libraries, but this is not really desirable.
• Must generate the dex file after proguard to make sure the mappings are the same. Gradle has no support to merge assets after Proguard (We want to put the dex files in the assets folder)

The other important chunk of code resides in SecondaryDex.java which essentially loads up the second dex file & injects the path of DEX file into the runtime class path. You can optimize this and just inject the path instead of reading the DEX file everytime the app is resumed.

I did the secondary Dex experiment on Google Play Services (which adds 20K methods) and was able to separate into a separate DEX file. This way my main dex file is unaffected by the bloat in Google Play services.

To understand how the Gradle task cycle works, you can refer to the BasePlugin.groovy source, you can see that it is difficult to control some aspects until there is a proper API for accessing variant objects and build tasks.

Answer 2

Update for Android Gradle plugin 2.2.0: It is not possible to access the dex task anymore, but in exchange additionalParameters was introduced as part of dexOptions. Use it like

android {
  dexOptions {
    additionalParameters += '--minimal-main-dex'
    // additionalParameters += '--main-dex-list=$projectDir/<filename>'.toString()
    // additionalParameters += '--set-max-idx-number=55000'
  }
}

Update for Android Gradle plugin 0.14.0: There is now direct multi-dex support via the new multiDexEnabled true directive (requires build-tools 21.1.0, support repository revision 8, and Android Studio 0.9).

Original answer: Ever since Gradle Android plugin 0.9.0 you actually can pass the --multi-dex to dx by adding this to you app's build.gradle file:

afterEvaluate {
    tasks.matching {
        it.name.startsWith('dex')
    }.each { dx ->
        if (dx.additionalParameters == null) {
            dx.additionalParameters = ['--multi-dex']
        } else {
            dx.additionalParameters += '--multi-dex'
        }

        // Add more additional parameters like this:
        dx.additionalParameters += '--main-dex-list=class-list.txt'
        dx.additionalParameters += '--minimal-main-dex'
    }
}

So far for the creating he multiple dex files. To actually use the multiple dex files, take a look at https://github.com/casidiablo/multidex (which is a fork of Google's upcoming MultiDex support library).

Answer 3

An example project partitioning and loading different dex files can be found here:

https://code.google.com/p/android-custom-class-loading-sample/

EDIT: For Gradle you already have an answer

Custom Class Loading in Dalvik with Gradle (Android New Build System)

Answer 4

In case gms was your issue and you are using gradle

Starting from gms version 6.5 you can choose individual API libraries

for example to include only the Maps API :

compile 'com.google.android.gms:play-services-maps:6.5.87'

and here is the complete list :

      com.google.android.gms:play-services-base:6.5.87
      com.google.android.gms:play-services-ads:6.5.87
      com.google.android.gms:play-services-appindexing:6.5.87
      com.google.android.gms:play-services-maps:6.5.87
      com.google.android.gms:play-services-location:6.5.87
      com.google.android.gms:play-services-fitness:6.5.87
      com.google.android.gms:play-services-panorama:6.5.87
      com.google.android.gms:play-services-drive:6.5.87
      com.google.android.gms:play-services-games:6.5.87
      com.google.android.gms:play-services-wallet:6.5.87
      com.google.android.gms:play-services-identity:6.5.87
      com.google.android.gms:play-services-cast:6.5.87
      com.google.android.gms:play-services-plus:6.5.87
      com.google.android.gms:play-services-appstate:6.5.87
      com.google.android.gms:play-services-wearable:6.5.87
      com.google.android.gms:play-services-all-wear:6.5.87