As hinted to by @LukasKörfer in a comment, to really remove a task from the build, instead of just skipping it, one solution is to add this to your build script:
project.gradle.startParameter.excludedTaskNames.add('yourTaskName')
However this seems to remove the task for all subprojects.