When I set the transform property of my CALayer with a CATransform3DRotate, the layer is properly rotated. However, the edges of the layer are jagged and not anti-aliased. I
As of iOS7, this is now finally possible on a per-layer basis and without any ugly hacks like transparent pixels or workarounds like rasterization:
layer.allowsEdgeAntialiasing = YES;
Interestingly, this is not covered by the official CALayer documentation, but is definitely a public API. See the iOS7 API diffs and Peter Steinberger's great write-up on Hidden Gems and Workarounds in iOS7.