It's a bit of both actually. Your browser will prevent CORS requests unless the origin of the request (i.e the referrer URL domain) is in a white list on the destination, or the destination approves all requests regardless of origin.
In both cases, the required header (Access-Control-Allow-Origin) is added which tells the browser that it's ok to send the request to the destination.
This ensures that people with malicious intent cannot send requests to another domain without the the user knowing about it.