I have a website which I support as far as IE8, no further.
When I first launched the site, I decided to use CSS vendor prefixes for CSSs elements such as border-rad
If you really want to get rid of prefixes, one of the solutions you can try is -prefix-free. It's a javascript plugin which loops through your stylesheets and, according to current browser removes the unused ones.
Although I didn't test it, I think it will definetely lower the performance.
You can also remove prefixes for properties which doesn't have a signifact meaning for functionality and/or user experience, like border-radius, box-shadow etc. You would have to test each element how it behaves without these properties. E.g. you have a button with border-radius: 4px. In a browser which doesn't support border-radius it will simply have rough corners. You must only consider if its worth sacrificing.