I\'m trying to set up a htaccess file that would accomplish the following:
Only allow my website to be viewed if the viewing user is coming from a specific domain
You are matching your referer against ^https://(.+\.)*mydomain\.com. Which means if some completely other site, say http://stealing_your_images.com/ links to something on protect.mydomain.com, the first condition will fail, thus the request is never redirected to https://unprotected.mydomain.com/. You want to approach it from the other direction, only allow certain referers to pass through, then redirect everything else:
RewriteEngine On
RewriteBase /
# allow these referers to passthrough
RewriteCond %{HTTP_REFERER} ^https://(protect|unprotected)\.mydomain\.com
RewriteRule ^ - [L]
# redirect everything else
RewriteRule ^ https://unprotected.mydomain.com/ [R,L]