When loaded a shared library is opened via the function dlopen(), is there a way for it to call functions in main program?
The dlopen() function, as discussed by @litb, is primarily provided on systems using ELF format object files. It is rather powerful and will let you control whether symbols referenced by the loaded library can be satisfied from the main program, and generally does let them be satisfied. Not all shared library loading systems are as flexible - be aware if it comes to porting your code.
The callback mechanism outlined by @hhafez works now that the kinks in that code are straightened out.