You can use @com.fasterxml.jackson.annotation.JsonIgnore annotation to avoid a field to be serialized. Thus the getter of that field won't get called.
@JsonIgnore
@OneToMany( mappedBy = "author", cascade = CascadeType.ALL, orphanRemoval = true)
private Set books = new HashSet();