We\'re currently using Matlab\'s fmincon function to do non-linear optimization for a project I\'m working on. We need to port that part of the project to C++ in o
Most of the time, libraries out there won't try to be a black box magic one-size-fits-all optimization tool like fmincon is- instead they'll require you to provide more detail and make more choices on your own, which is simpler for them and should result in your software being faster. You can certainly use the MATLAB Engine or MATLAB Compiler to call fmincon from your program, but most likely your software will run a good deal faster (and you can avoid purchasing the MATLAB Compiler) if you can use a little more knowledge about the structure your optimization problem has and call an appropriate algorithm.
Your background info doesn't describe what you're doing - esp. what your feasible set is- clearly enough for me to be able to tell you what to use, so all I can do is point you in the direction of relevant resources.
Wikipedia's page on optimization links to lists of optimization software- most importantly, it describes more specific kinds of optimization problems (for instance, can you formulate your problem as quadratic programming with linear constraints?) and software appropriate for each situation.
Boyd's book on convex optimization and the linked course materials & videos are really good resources.