If I want my app to behave differently on a jailbroken iPhone, how would I go about determining this?
I am not aware of any "APIs" that exist for this. If there were, then a jailbreak-masking product would quickly cover them up.
As lots of people point out, it is a cat-and-mouse game. And after both players become expert, it all comes down to who gets the first move. (Person holding the device.)
I found many good suggestions for detecting jailbreak in Zdziarski's new book "Hacking and Securing iOS Apps". (Personally, I paid more for the O'Reilly eBook because they permit copy-and-paste.)
No, I am not affiliated with the publishers. But I did find it a good book. I don't like to just publish hackers' mistakes so they can fix them, so I thought I'd point to the book.