Eventually WCF duplex Silverlight 4 client start getting 404 Not Found
errors for poll messages, immediately after poll was send from WCF service to Silverlight
If everything seems to be working fine, then perhaps this is a network infrastructure/configuration issue (e.g. dns configuration). Do you get the same issue when running locally or using ip address instead of host name?
Similar issue may also occur if you have several bindings configured on Site in IIS (see here for details: http://blogs.msdn.com/b/rampo/archive/2008/02/11/how-can-wcf-support-multiple-iis-binding-specified-per-site.aspx)
Another thing is how do you communicated from server to clients. If you iterate clients in a loop and call callback methods one by one, then you may get timeouts which would show as 404s. Callback generally should be invoked on background threads instead (one per client).
Depending on how you do the communication, this may also be caused by the deadlock (when UI thread is involved in sending/receiving messages/callbacks to and from service).