I was reading through the specification and many examples about usage of websockets. Almost all of them talk about UTF-8 or ascii message transfer using websockets.
It may be a while before non-UTF-8 (i.e. binary) encoding is supported in WebSockets.
I suggest using base64 encode/decode on the client and server. All browsers with WebSockets support have window.atob (base64 decode) and window.btoa (base64 encode). Most languages you might write a WebSockets server in have base64 libraries (i.e. base64 module in python).
If you are wanting to transfer binary data you might be interesting in wsproxy included with noVNC which is a web based VNC client. wsproxy (there is a C and python version) is a WebSockets to generic TCP sockets proxy. It base64 encodes/decodes all traffic to/from the browser. You can use it to connect from a WebSockets capable browser to any type of TCP port.
Disclaimer: I created noVNC.