The fixed point combinator doesn\'t always produce the right answer given the definition:
fix f = f (fix f)
The following code does not ter
Fixed point combinator finds the least-defined fixed point of a function, which is ⊥ in your case (non-termination indeed is undefined value).
You can check, that in your case
(\x -> x * x) ⊥ = ⊥
i.e. ⊥ really is fixed point of \x -> x * x.
As for why is fix defined that way: the main point of fix is to allow you use anonymous recursion and for that you do not need more sophisticated definition.