How would the condition be written to ensure a page is either accessed by xmlhttp request from my site or from an allowed outside domain?
Considering that both Referer and X-Request-With headers are sent (or not sent) by the client (the browser, or anything else that can send an HTTP request), they cannot be trusted.
You can use those as hints, to enhance user-experience ; but you must not rely on them to be either present or correct.
Basically, you have no way to be sure that a request comes from a specific domain (even for XmlHttpRequest : the browser can only use XHR on the same domain... But you have no way to be sure that a request you receive is, or is not, coming from XHR).
Amongst possible ideas (not sure what your real problem / need is), you might try using some kind of API-key, to limit request-rates or so ?