I am trying to solve this recurrence
T(n) = 3 T(n/2) + n lg n ..
I have come to the solution that it belongs to masters theorem case 2 since n lg n is O(n^2
n*log(n) is not O(n^2). It's known as quasi-linear and it grows much slower than O(n^2). In fact n*log(n) is less than polynomial.
In other words:
O(n*log(n)) < O(n^k)
where k > 1
In your example:
3*T(2n) -> O(n^1.585)
Since O(n^1.585) is polynomial and dominates O(n*log(n)), the latter term drops off so the final complexity is just O(n^1.585).