It seems that fieldset defaults to 100% width of its container. Is there any way that you can have the field set just be as big as the widest control inside the fieldset?
Use display: inline-block, though you need to wrap it inside a DIV to keep it from actually displaying inline. Tested in Safari.
display: inline-block
Blah ...