Assuming arbitrary content, you cannot avoid a worst-case of O(n2).* You have to visit every element in each column that you want to consider, and in the worst-case, you have to consider all columns.
* Also assuming that n is the matrix dimension here, not the total number of elements.