I wonder if this is possible to tell via .htaccess or any other manners to set a base URL.
Example
My CSS in the index.php is set like this
What you might be looking for is the HTML tag. It's the base URI of the document, all relative links are resolved to. If the tag is not within the documents source, the URI from the browser address will be used.
Instead of this you can use absolute URIs as well.
Please see Links in HTML Documents to understand what you're making use of.