Opencv program becomes much slower when compiled with the same lib in a new directory

Question

I wrote a program using Opencv. The program goes correctly but for some reasons, I have to move the Opencv dependencies from the system directory, i.e. /usr/local/lib/