I am trying to detect whether my process is being run in a debugger or not and, while in Windows there are many solutions and in Linux I use:
ptrace(PTRACE_ME,0
#include
#include
#include
static bool amIAnInferior(void)
{
mach_msg_type_number_t count = 0;
exception_mask_t masks[EXC_TYPES_COUNT];
mach_port_t ports[EXC_TYPES_COUNT];
exception_behavior_t behaviors[EXC_TYPES_COUNT];
thread_state_flavor_t flavors[EXC_TYPES_COUNT];
exception_mask_t mask = EXC_MASK_ALL & ~(EXC_MASK_RESOURCE | EXC_MASK_GUARD);
kern_return_t result = task_get_exception_ports(mach_task_self(), mask, masks, &count, ports, behaviors, flavors);
if (result == KERN_SUCCESS)
{
for (mach_msg_type_number_t portIndex = 0; portIndex < count; portIndex++)
{
if (MACH_PORT_VALID(ports[portIndex]))
{
return true;
}
}
}
return false;
}
This looks and sees if there is an active exception handler in our process (for EXC_BREAKPOINT, EXC_BAD_ACCESS, etc). Ptrace is not required to achieve this in a debugger, thus relying only on a ptrace flag to be set is not quite ideal.
This approach is mentioned in http://reverse.put.as/wp-content/uploads/2012/07/Secuinside-2012-Presentation.pdf
My blog post describes this in more detail.