Does anybody know how to remove the dotted outline on buttons in Opera?
This is less of an answer, and more of an explanation as to what seems to be going on:
My reason for removing opera's outline was to add an outline of my own. To add an outline I used:
:focus{
outline:1px dotted #999;
outline-offset:-3px;
}
This works perfectly fine in every other browser... except Opera. Opera instead gives you this weird interference pattern, which looks like a dotted-dashed outline:

now if you remove your outline, you still have the standard outline that Opera provides, a nice simple 1px spaced dotted line:

Since I have no way of adding a style to every browser except Opera, I instead decided on removing Opera's outline before adding my own. Using brothercake's solution, -o-transform:rotate(0); to do this and then applying my own outline:

Voila!
From what I can tell, Opera adds it's own secondary outline on top of any outline defined by CSS.
This secondary outline seems to have an independent color, width, style, and offset.
Color is the opposite of the background,
Width is 1px,
Style is dotted,
and the offset is 2px smaller than the border.

sorry there is no style image, the upload didn't work correctly
one interesting thing is that the dotted style of the Opera outline is not the same as the CSS outline's dotted, otherwise there would be no interference pattern with both:

As I stated above, I am using brothercake's solution which is to nullify the opera border with:
-o-transform:rotate(0);
As he mentioned in his later comment this 'fix' does have some issues as it is a rendering bug:
I have noticed that when returning window or tab focus to the page containing the button, if the button previously had focus, the Opera outline will reappears until the button loses focus or is hovered over.