Remove columns in which factors appear only once in a matrix column in R

Question

I have a matrix with different factors in columns. I want to drop columns in which factors appear only once in a column in R.

My example matrix is:

n.m