I have written a code in C++ which uses boost library and also uses multiple files. I am trying to execute the function in R.
Here is my C++ function prototype:
Since this a multi-file C++ project, you should package the code as an R package. It is actually quite easy to do using Rcpp, see for example this answer. The Rcpp-package vignette contains further information. In addition, an easy way to make use of boost is via the BH package. There are many example packages for using Rcpp + BH on CRAN which you can study, e.g. my own dqrng. Some more details can also be found in the R Packages book and of course in the official documentation Writing R Extensions.
The crucial thing for using the BH package is LinkingTo: BH in the DESCRIPTION file. This ensures that -I is part of the compiler flags. In order to use header files that you place in inst/include, you should add PKG_CPPFLAGS = -I../inst/include to src/Makevars.