Converting & validating CSV file upload in Spring MVC

问题

I have a Customer entity that contains a list of Sites, as follows:

public class Customer {

    @Id
    @GeneratedValue
    private int id;

    @NotNull
    private String name;

    @NotNull
    @AccountNumber
    private String accountNumber;

    @Valid
    @OneToMany(mappedBy="customer")
    private List<Site> sites
}

public class Site {

    @Id
    @GeneratedValue
    private int id;

    @NotNull
    private String addressLine1;

    private String addressLine2;

    @NotNull
    private String town;

    @PostCode
    private String postCode;

    @ManyToOne
    @JoinColumn(name="customer_id")
    private Customer customer;
}

I am in the process of creating a form to allow users to create a new Customer by entering the name & account number and supplying a CSV file of sites (in the format "addressLine1", "addressLine2", "town", "postCode"). The user's input needs to be validated and errors returned to them (e.g. "file is not CSV file", "problem on line 7").

I started off by creating a Converter to receive a MultipartFile and convert it into a list of Site:

public class CSVToSiteConverter implements Converter<MultipartFile, List<Site>> {

    public List<Site> convert(MultipartFile csvFile) {

        List<Site> results = new List<Site>();

        /* open MultipartFile and loop through line-by-line, adding into List<Site> */

        return results;
    }
}

This worked but there is no validation (i.e. if the user uploads a binary file or one of the CSV rows doesn't contain a town), there doesn't seem to be a way to pass the error back (and the converter doesn't seem to be the right place to perform validation).

I then created a form-backing object to receive the MultipartFile and Customer, and put validation on the MultipartFile:

public class CustomerForm {

    @Valid
    private Customer customer;

    @SiteCSVFile
    private MultipartFile csvFile;
}

@Documented
@Constraint(validatedBy = SiteCSVFileValidator.class)
@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
public @interface SiteCSVFile {

    String message() default "{SiteCSVFile}";

    Class<?>[] groups() default {};

    Class<? extends Payload>[] payload() default {};
}

public class SiteCSVFileValidator implements ConstraintValidator<SiteCSVFile, MultipartFile> {

    @Override
    public void initialize(SiteCSVFile siteCSVFile) { }

    @Override
    public boolean isValid(MultipartFile csvFile, ConstraintValidatorContext cxt) {

        boolean wasValid = true;

        /* test csvFile for mimetype, open and loop through line-by-line, validating number of columns etc. */

        return wasValid;
    }
}

This also worked but then I have to re-open the CSV file and loop through it to actually populate the List within Customer, which doesn't seem that elegant:

@RequestMapping(value="/new", method = RequestMethod.POST)
public String newCustomer(@Valid @ModelAttribute("customerForm") CustomerForm customerForm, BindingResult bindingResult) {

    if (bindingResult.hasErrors()) {
        return "NewCustomer";
    } else {

        /* 
           validation has passed, so now we must:
           1) open customerForm.csvFile 
           2) loop through it to populate customerForm.customer.sites 
        */

        customerService.insert(customerForm.customer);

        return "CustomerList";
    }
}

My MVC config limits file uploads to 1MB:

@Bean
public MultipartResolver multipartResolver() {
    CommonsMultipartResolver multipartResolver = new CommonsMultipartResolver();
    multipartResolver.setMaxUploadSize(1000000);
    return multipartResolver;
}

Is there a spring-way of converting AND validating at the same time, without having to open the CSV file and loop through it twice, once to validate and another to actually read/populate the data?

回答1:

IMHO, it is a bad idea to load the whole CSV in memory unless :

you are sure it will always be very small (and what if a user click on wrong file ?)
the validation is global (only real use case, but does not seem to be here)
your application will never be used in a production context under serious load

You should either stick to the MultipartFile object, or use a wrapper exposing the InputStream (and eventually other informations you could need) if you do not want to tie your business classes to Spring.

Then you carefully design, code and test a method taking an InputStream as input, reads it line by line and call line by line methods to validate and insert data. Something like

class CsvLoader {
@Autowired Verifier verifier;
@Autowired Loader loader;

    void verifAndLoad(InputStream csv) {
        // loop through csv
        if (verifier.verify(myObj)) {
            loader.load(myObj);
        }
        else {
            // log the problem eventually store the line for further analysis
        }
        csv.close();
    }
}

That way, your application only uses the memory it really needs, only looping once other the file.

Edit : precisions on what I meant by wrapping Spring MultipartFile

First, I would split validation in 2. Formal validation is in controller layer and only controls that :

there is a Customer field
the file size and mimetype seems Ok (eg : size > 12 && mimetype = text/csv)

The validation of the content is IMHO a business layer validation and can happen later. In this pattern, SiteCSVFileValidator would only test csv for mimetype and size.

Normally, you avoid directly using Spring classes from business classes. If it is not a concern, the controller directly sends the MultipartFile to a service object, passing also the BindingResult to populate directly the eventual error messages. The controller becomes :

@RequestMapping(value="/new", method = RequestMethod.POST)
public String newCustomer(@Valid @ModelAttribute("customerForm") CustomerForm customerForm, BindingResult bindingResult) {

    if (bindingResult.hasErrors()) {
        return "NewCustomer"; // only external validation
    } else {

        /* 
           validation has passed, so now we must:
           1) open customerForm.csvFile 
           2) loop through it to validate each line and populate customerForm.customer.sites 
        */

        customerService.insert(customerForm.customer, customerForm.csvFile, bindingResult);
        if (bindingResult.hasErrors()) {
            return "NewCustomer"; // only external validation
        } else {
            return "CustomerList";
        }
    }
}

In service class we have

insert(Customer customer, MultipartFile csvFile, Errors errors) {
    // loop through csvFile.getInputStream populating customer.sites and eventually adding Errors to errors
    if (! errors.hasErrors) {
        // actually insert through DAO
    }
}

But we get 2 Spring classes in a method of service layer. If it is a concern, just replace the line customerService.insert(customerForm.customer, customerForm.csvFile, bindingResult); with :

List<Integer> linesInError = new ArrayList<Integer>();
customerService.insert(customerForm.customer, customerForm.csvFile.getInputStream(), linesInError);
if (! linesInError.isEmpty()) {
    // populates bindingResult with convenient error messages
}

Then the service class only adds line numbers where errors where detected to linesInError but it only gets the InputStream, where it could need say the original file name. You can pass the name as another parameter, or use a wrapper class :

class CsvFile {

    private String name;
    private InputStream inputStream;

    CsvFile(MultipartFile file) {
        name = file.getOriginalFilename();
        inputStream = file.getInputStream();
    }
    // public getters ...
}

and call

customerService.insert(customerForm.customer, new CsvFile(customerForm.csvFile), linesInError);

with no direct Spring dependancies

来源：https://stackoverflow.com/questions/25460779/converting-validating-csv-file-upload-in-spring-mvc

标签

java

validation

spring-mvc

csv