I believe this is probably faster than any other way if you don't care about the lists order, if you do take care about the final order store the indexes from the original and resort by that.
category_ids.sort()
ones_last_index = category_ids.count('1')
del category_ids[0:ones_last_index]