I am going to have an ASP.net MVC web site (example.com) and a Web API site (api.example.com) running on the same domain. What is the best and most secure way to use Forms A
Take a look at this link http://www.codeproject.com/Articles/27576/Single-Sign-on-in-ASP-NET-and-Other-Platforms this covers the answer in detail.
You will need to ensure all machines and separate applications on the site share a common (but unique to production) machine key to allow the authentication cookies to be trusted by all the machines/applications.
If you are simply using virtual directories under the same sub domain then simply harmonising the web.conig Forms Auth settings and machine keys should get you up and running very quickly.
If you want this to work between a second level domain then you need to change the "Domain" setting on the Form's Auth cookie. See the article for details.