Using some help I've found the answer, which was a combination of 2 missing pieces (one of which was referred to in the comments):
Need to set this:
System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true");
Must set the "endPoint" (which was not required for upload or download):
s3Client.setEndpoint(endpoint);
Optionally it might be useful to also add this:
s3Client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));