Well I had to deal with something similar. My Computer is in a restricted domain, so I don't have administrator privileges. After some research and reading I found this thread and the netsh hints made me use temporary acl bindings just for developing tests. On my computer these rule exists.
There's this entry:
Run 'netsh http show urlacl' (as shown above)
[...]
Reservierte URL : http://+:80/Temporary_Listen_Addresses/
Benutzer: \Jeder
Abhören: Yes
Delegieren: No
SDDL: D:(A;;GX;;;WD)
[...]
So I can use the HttpListener as non-admin (Jeder):
[...]
HttpListener l = new HttpListener();
string prefix = "http://+:80/Temporary_Listen_Addresses/";
l.Prefixes.Add(prefix);
l.Start(); // does not throw any "Permission Denied/Access Denied/Zugriff verweigert"
[...]
May this helps anybody finding this thread.