In my WCF service, when trying transfer large data I constantly get an error: The underlying connection was closed: The connection was closed unexpectedly>
Ignoring the problems with the maxRequestLength (which have been answered by others), I will have a go at answering your original question about how to troubleshoot WCF.
If you are already using the Service Trace Viewer (I couldn't tell from the question if you were just viewing them by hand) - it is possible that all the details aren't going into the file.
When I want to get really hardcore, I enable all the logging parameters for message logging. (This will generate some big service logs though so don't leave it on)
If you are not using the Microsoft Service Trace Viewer I recommend that. It gives all the information I need to track down those tricky message handshake, message size exceptions etc. Here is an MSDN reference to get you started http://msdn.microsoft.com/en-us/library/aa751795.aspx
Trace interactions that have potential problems are hilighted in yellow on the left, and the detailed pane on the top right will normally hilight the exceptional service event in red. Sometimes you will get multiple problems as the inner error cascades through the service stack - but you can see it all in the trace viewer.

If you get nothing in your server 'service log', then it is possible your exceptions are entirely at the client end - theoretically you could exceed some of the client side security parameters (message size etc) before any message has actually reached the web service end - but client problems are generally easier to track down because you know you only have to worry about editing the config file at the client end (i.e. it's not because of any interaction between the client and server settings).