In a let form (Clojure here) I can doing something like
(let [[u s v] (svd A)]
(do-something-with u v))
where svd
What follows are some philosophical justifications.
Clojure favors immutability over mutability, and all sources of mutability should be carefully considered and named. def creates mutable vars. Idiomatic Clojure therefore both doesn't use them much anyway, and also would not want it to be too easy to create many mutable vars without care (e.g. by destructuring). let and function argument destructuring, however, creates immutable bindings, so Clojure makes those bindings easy to create.
Vars created by def have global scope. Therefore you should name defed vars carefully and keep them few in number. Destructuring def would make it too easy to create many defs without care. let and function argument destructuring, on the other hand, creates local, lexically-scoped bindings, so the convenience of destructuring does not cause name pollution.