For posterity, here's the strategy to get around this problem in our automated build script:
- Before building, move the ~/Library/MobileDevices/Provisioning Profiles directory aside
- Check to see if the iPhone Configuration Utility is running and close it (certs always reappear if this app is open. In XCode 4.1 you need to close the organizer too.)
- Run you build
- Move the Provisioning Profiles directory back
- Profit?