I know your question is specific to using the gradle wrapper, but if you're using the Gradle Tooling API, you can control the color output via
setColorOutput. Combined with the --quiet argument (set via withArguments), this results in plain-text output.