Java 9 HttpClient send a multipart/form-data request

Question

Below is a form:

Answer 1

I recently released a library providing useful extensions to Java 11's HttpClient. The library contains a MultipartBodyPublisher with a convenient and easy to use MultipartBodyPublisher.Builder. Here is an example using it (JDK11 or later is required):

MultipartBodyPublisher multipartBody = MultipartBodyPublisher.newBuilder()
    .textPart("foo", "foo_text")
    .filePart("bar", Path.of("path/to/file.txt"))
    .formPart("baz", BodyPublishers.ofInputStream(() -> ...))
    .build();
HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://example.com/"))
    .POST(multipartBody)
    .build();

Note that you can also add any BodyPublisher (or HttpHeaders) you want. See the user guide for more details.

Answer 2

It is possible to use multipart/form-data or any other content type - but you have to encode the body in the correct format yourself. The client itself does not do any encoding based on the content type.

That means your best option is to use another HTTP client the like Apache HttpComponents client or only use the encoder of another library like in the example of @nullpointer's answer.

---

If you do encode the body yourself, note that you can't call methods like POST more than once. POST simply sets the BodyProcessor and calling it again will just override any previously set processors. You have to implement one processor that produces the whole body in the correct format.

For multipart/form-data that means:

1. Set the boundary header to an appropriate value

2. Encode each parameter so that it looks like in your example. Basically something like this for text input:

   boundary + "\nContent-Disposition: form-data; name=\"" + name + "\"\n\n" + value + "\n"
   

   Here, the name refers to the name attribute in the HTML form. For the file input in the question, this would img and the value would be the encoded file content.

Answer 3

While the correct answer is full-blown implementation and might be correct, it did not work for me.

My solution took inspiration from here. I just cleaned up for my use case not required parts. Me, personally, use multipart form for only uploading picture or zip file (singular). The code:

    public static HttpRequest buildMultiformRequest(byte[] body) {
        String boundary = "-------------" + UUID.randomUUID().toString();
        Map<String, byte[]> data = Map.of("formFile", body);

        return HttpRequest.newBuilder()
                .uri(URI.create(<URL>))
                .POST(HttpRequest.BodyPublishers.ofByteArrays(buildMultipartData(data, boundary, "filename.jpeg", MediaType.IMAGE_JPEG_VALUE)))
                .header("Content-Type", "multipart/form-data; boundary=" + boundary)
                .header("Accept", MediaType.APPLICATION_JSON_VALUE)
                .timeout(Duration.of(5, ChronoUnit.SECONDS))
                .build();
    }

    public static ArrayList<byte[]> buildMultipartData(Map<String, byte[]> data, String boundary, String filename, String mediaType) {
        var byteArrays = new ArrayList<byte[]>();
        var separator = ("--" + boundary + "\r\nContent-Disposition: form-data; name=").getBytes(StandardCharsets.UTF_8);

        for (var entry : data.entrySet()) {
            byteArrays.add(separator);
            byteArrays.add(("\"" + entry.getKey() + "\"; filename=\"" + filename + "\"\r\nContent-Type:" + mediaType + "\r\n\r\n").getBytes(StandardCharsets.UTF_8));
            byteArrays.add(entry.getValue());
            byteArrays.add("\r\n".getBytes(StandardCharsets.UTF_8));
        }

        byteArrays.add(("--" + boundary + "--").getBytes(StandardCharsets.UTF_8));
        return byteArrays;
    }

Answer 4

A direction in which you can attain making a multiform-data call could be as follows:

BodyProcessor can be used with their default implementations or else a custom implementation can also be used. Few of the ways to use them are :

1. Read the processor via a string as :

   HttpRequest.BodyProcessor dataProcessor = HttpRequest.BodyProcessor.fromString("{\"username\":\"foo\"}")
   

2. Creating a processor from a file using its path

   Path path = Paths.get("/path/to/your/file"); // in your case path to 'img'
   HttpRequest.BodyProcessor fileProcessor = HttpRequest.BodyProcessor.fromFile(path);
   

OR

3. You can convert the file input to a byte array using the apache.commons.lang(or a custom method you can come up with) to add a small util like :

   org.apache.commons.fileupload.FileItem file;
   
   org.apache.http.HttpEntity multipartEntity = org.apache.http.entity.mime.MultipartEntityBuilder.create()
          .addPart("username",new StringBody("foo", Charset.forName("utf-8")))
          .addPart("img", newFileBody(file))
          .build();
   multipartEntity.writeTo(byteArrayOutputStream);
   byte[] bytes = byteArrayOutputStream.toByteArray();
   

   and then the byte[] can be used with BodyProcessor as:

   HttpRequest.BodyProcessor byteProcessor = HttpRequest.BodyProcessor.fromByteArray();
   

---

Further, you can create the request as :

HttpRequest request = HttpRequest.newBuilder()
            .uri(new URI("http:///example/html5/demo_form.asp"))
            .headers("Content-Type","multipart/form-data","boundary","boundaryValue") // appropriate boundary values
            .POST(dataProcessor)
            .POST(fileProcessor)
            .POST(byteProcessor) //self-sufficient
            .build();

---

The response for the same can be handled as a file and with a new HttpClient using

HttpResponse.BodyHandler bodyHandler = HttpResponse.BodyHandler.asFile(Paths.get("/path"));

HttpClient client = HttpClient.newBuilder().build();

as:

HttpResponse response = client.send(request, bodyHandler);
System.out.println(response.body());

Answer 5

I wanted to do this for a project without having to pull in the Apache client, so I wrote a MultiPartBodyPublisher (Java 11, fyi):

import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.net.http.HttpRequest;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import java.util.function.Supplier;

public class MultiPartBodyPublisher {
    private List<PartsSpecification> partsSpecificationList = new ArrayList<>();
    private String boundary = UUID.randomUUID().toString();

    public HttpRequest.BodyPublisher build() {
        if (partsSpecificationList.size() == 0) {
            throw new IllegalStateException("Must have at least one part to build multipart message.");
        }
        addFinalBoundaryPart();
        return HttpRequest.BodyPublishers.ofByteArrays(PartsIterator::new);
    }

    public String getBoundary() {
        return boundary;
    }

    public MultiPartBodyPublisher addPart(String name, String value) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.STRING;
        newPart.name = name;
        newPart.value = value;
        partsSpecificationList.add(newPart);
        return this;
    }

    public MultiPartBodyPublisher addPart(String name, Path value) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.FILE;
        newPart.name = name;
        newPart.path = value;
        partsSpecificationList.add(newPart);
        return this;
    }

    public MultiPartBodyPublisher addPart(String name, Supplier<InputStream> value, String filename, String contentType) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.STREAM;
        newPart.name = name;
        newPart.stream = value;
        newPart.filename = filename;
        newPart.contentType = contentType;
        partsSpecificationList.add(newPart);
        return this;
    }

    private void addFinalBoundaryPart() {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.FINAL_BOUNDARY;
        newPart.value = "--" + boundary + "--";
        partsSpecificationList.add(newPart);
    }

    static class PartsSpecification {

        public enum TYPE {
            STRING, FILE, STREAM, FINAL_BOUNDARY
        }

        PartsSpecification.TYPE type;
        String name;
        String value;
        Path path;
        Supplier<InputStream> stream;
        String filename;
        String contentType;

    }

    class PartsIterator implements Iterator<byte[]> {

        private Iterator<PartsSpecification> iter;
        private InputStream currentFileInput;

        private boolean done;
        private byte[] next;

        PartsIterator() {
            iter = partsSpecificationList.iterator();
        }

        @Override
        public boolean hasNext() {
            if (done) return false;
            if (next != null) return true;
            try {
                next = computeNext();
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
            if (next == null) {
                done = true;
                return false;
            }
            return true;
        }

        @Override
        public byte[] next() {
            if (!hasNext()) throw new NoSuchElementException();
            byte[] res = next;
            next = null;
            return res;
        }

        private byte[] computeNext() throws IOException {
            if (currentFileInput == null) {
                if (!iter.hasNext()) return null;
                PartsSpecification nextPart = iter.next();
                if (PartsSpecification.TYPE.STRING.equals(nextPart.type)) {
                    String part =
                            "--" + boundary + "\r\n" +
                            "Content-Disposition: form-data; name=" + nextPart.name + "\r\n" +
                            "Content-Type: text/plain; charset=UTF-8\r\n\r\n" +
                            nextPart.value + "\r\n";
                    return part.getBytes(StandardCharsets.UTF_8);
                }
                if (PartsSpecification.TYPE.FINAL_BOUNDARY.equals(nextPart.type)) {
                    return nextPart.value.getBytes(StandardCharsets.UTF_8);
                }
                String filename;
                String contentType;
                if (PartsSpecification.TYPE.FILE.equals(nextPart.type)) {
                    Path path = nextPart.path;
                    filename = path.getFileName().toString();
                    contentType = Files.probeContentType(path);
                    if (contentType == null) contentType = "application/octet-stream";
                    currentFileInput = Files.newInputStream(path);
                } else {
                    filename = nextPart.filename;
                    contentType = nextPart.contentType;
                    if (contentType == null) contentType = "application/octet-stream";
                    currentFileInput = nextPart.stream.get();
                }
                String partHeader =
                        "--" + boundary + "\r\n" +
                        "Content-Disposition: form-data; name=" + nextPart.name + "; filename=" + filename + "\r\n" +
                        "Content-Type: " + contentType + "\r\n\r\n";
                return partHeader.getBytes(StandardCharsets.UTF_8);
            } else {
                byte[] buf = new byte[8192];
                int r = currentFileInput.read(buf);
                if (r > 0) {
                    byte[] actualBytes = new byte[r];
                    System.arraycopy(buf, 0, actualBytes, 0, r);
                    return actualBytes;
                } else {
                    currentFileInput.close();
                    currentFileInput = null;
                    return "\r\n".getBytes(StandardCharsets.UTF_8);
                }
            }
        }
    }
}

You can use it approximately like so:

MultiPartBodyPublisher publisher = new MultiPartBodyPublisher()
       .addPart("someString", "foo")
       .addPart("someInputStream", () -> this.getClass().getResourceAsStream("test.txt"), "test.txt", "text/plain")
       .addPart("someFile", pathObject);
HttpRequest request = HttpRequest.newBuilder()
       .uri(URI.create("https://www.example.com/dosomething"))
       .header("Content-Type", "multipart/form-data; boundary=" + publisher.getBoundary())
       .timeout(Duration.ofMinutes(1))
       .POST(publisher.build())
       .build();

Note that addPart for input streams actually takes a Supplier<InputStream> and not just an InputStream.

Answer 6

I struggled with this problem for a while, even after seeing and reading this page. But, using the answers on this page to point me in the right direction, reading more about multipart forms and boundaries, and tinkering around, I was able to create a working solution.

The gist of the solution is to use Apache's MultipartEntityBuilder to create the entity and its boundaries (HttpExceptionBuilder is a homegrown class):

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Optional;
import java.util.function.Supplier;

import org.apache.commons.lang3.Validate;
import org.apache.http.HttpEntity;
import org.apache.http.entity.BufferedHttpEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;

/**
 * Class containing static helper methods pertaining to HTTP interactions.
 */
public class HttpUtils {
    public static final String MULTIPART_FORM_DATA_BOUNDARY = "ThisIsMyBoundaryThereAreManyLikeItButThisOneIsMine";

    /**
     * Creates an {@link HttpEntity} from a {@link File}, loading it into a {@link BufferedHttpEntity}.
     *
     * @param file     the {@link File} from which to create an {@link HttpEntity}
     * @param partName an {@link Optional} denoting the name of the form data; defaults to {@code data}
     * @return an {@link HttpEntity} containing the contents of the provided {@code file}
     * @throws NullPointerException  if {@code file} or {@code partName} is null
     * @throws IllegalStateException if {@code file} does not exist
     * @throws HttpException         if file cannot be found or {@link FileInputStream} cannot be created
     */
    public static HttpEntity getFileAsBufferedMultipartEntity(final File file, final Optional<String> partName) {
        Validate.notNull(file, "file cannot be null");
        Validate.validState(file.exists(), "file must exist");
        Validate.notNull(partName, "partName cannot be null");

        final HttpEntity entity;
        final BufferedHttpEntity bufferedHttpEntity;

        try (final FileInputStream fis = new FileInputStream(file);
                final BufferedInputStream bis = new BufferedInputStream(fis)) {
            entity = MultipartEntityBuilder.create().setBoundary(MULTIPART_FORM_DATA_BOUNDARY)
                    .addBinaryBody(partName.orElse("data"), bis, ContentType.APPLICATION_OCTET_STREAM, file.getName())
                    .setContentType(ContentType.MULTIPART_FORM_DATA).build();

            try {
                bufferedHttpEntity = new BufferedHttpEntity(entity);
            } catch (final IOException e) {
                throw HttpExceptionBuilder.create().withMessage("Unable to create BufferedHttpEntity").withThrowable(e)
                        .build();
            }
        } catch (final FileNotFoundException e) {
            throw HttpExceptionBuilder.create()
                    .withMessage("File does not exist or is not readable: %s", file.getAbsolutePath()).withThrowable(e)
                    .build();
        } catch (final IOException e) {
            throw HttpExceptionBuilder.create()
                    .withMessage("Unable to create multipart entity from file: %s", file.getAbsolutePath())
                    .withThrowable(e).build();
        }

        return bufferedHttpEntity;
    }

    /**
     * Returns a {@link Supplier} of {@link InputStream} containing the content of the provided {@link HttpEntity}. This
     * method closes the {@code InputStream}.
     *
     * @param entity the {@link HttpEntity} from which to get an {@link InputStream}
     * @return an {@link InputStream} containing the {@link HttpEntity#getContent() content}
     * @throws NullPointerException if {@code entity} is null
     * @throws HttpException        if something goes wrong
     */
    public static Supplier<? extends InputStream> getInputStreamFromHttpEntity(final HttpEntity entity) {
        Validate.notNull(entity, "entity cannot be null");

        return () -> {
            try (final InputStream is = entity.getContent()) {
                return is;
            } catch (final UnsupportedOperationException | IOException e) {
                throw HttpExceptionBuilder.create().withMessage("Unable to get InputStream from HttpEntity")
                        .withThrowable(e).build();
            }
        };
    }
}

And then a method that uses these helper methods:

private String doUpload(final File uploadFile, final String filePostUrl) {
    assert uploadFile != null : "uploadFile cannot be null";
    assert uploadFile.exists() : "uploadFile must exist";
    assert StringUtils.notBlank(filePostUrl, "filePostUrl cannot be blank");

    final URI uri = URI.create(filePostUrl);
    final HttpEntity entity = HttpUtils.getFileAsBufferedMultipartEntity(uploadFile, Optional.of("partName"));
    final String response;

    try {
        final Builder requestBuilder = HttpRequest.newBuilder(uri)
                .POST(BodyPublisher.fromInputStream(HttpUtils.getInputStreamFromHttpEntity(entity)))
                .header("Content-Type", "multipart/form-data; boundary=" + HttpUtils.MULTIPART_FORM_DATA_BOUNDARY);

        response = this.httpClient.send(requestBuilder.build(), BodyHandler.asString());
    } catch (InterruptedException | ExecutionException e) {
        throw HttpExceptionBuilder.create().withMessage("Unable to get InputStream from HttpEntity")
                    .withThrowable(e).build();
    }

    LOGGER.info("Http Response: {}", response);
    return response;
}