Regarding the core limit, you can do it yourself in C by calling setrlimit.
On a GNU (glibc) or BSD system, you can get a backtrace by calling backtrace and related system calls. You will then have to translate the function addresses into function names by running addr2line (or duplicating its functionality).
Just don't use -g, you can still get a backtrace (except that inlined functions will not appear).