It's not that they're 5% of your user base, they're 5% of your IE user base. In otherwords, IE6 accounts for 5% of 30% of your traffic.
I would say no, because you'd be sacrificing too much time and features for too few people. That and big websites like youtube and firefox have dropped support already.
