If I want my app to behave differently on a jailbroken iPhone, how would I go about determining this?
Checking if the kernel is broken isn't THAT much more involved.
Jailbreaking makes the kernel's signature check of signed code always report that code is signed correctly, unbroken phones cannot run code with a bad signature.
So, include a separate executable in the app with a bad signature. It could just be a 3-line program that has main() and a return value. Compile the executable without code signing (turn it off in Project Settings->Build) and sign it with a different key using the "codesign" commandline utility.
Have your app exec the separate executable. If your program can't get the return value when running the separate executable with the bad sig, it's definitely jailed. If the separate executable returns A-OK, the phone is definitely jailbroken.