I wonder if this is possible to tell via .htaccess or any other manners to set a base URL.
Example
My CSS in the index.php is set like this
You're looking for the html base tag. To quote the docs:
For example, given the following BASE declaration and A declaration:
Our Products Have you seen our Bird Cages?
the relative URI "../cages/birds.gif" would resolve to:
http://www.aviary.com/cages/birds.gif