I have an application including two libraries, where both of them have dependencies on native libraries. Both are included using gradle so the structure looks like this:
https://stackoverflow.com/a/30799825/3325222 worked for me in a similar situation. However, that solution is actually deprecated.
There is new SDK support in experimental Gradle 2.5 which supports architecture filters: http://tools.android.com/tech-docs/new-build-system/tips
I found a solution by excluding the specific *.so files that were just added inside the architectures i didn't want to support:
packagingOptions {
exclude 'lib/arm64-v8a/lib.so'
exclude 'lib/mips/lib.so'
}