How to parse and validate a WebSocket frame in Java?

Question

I wrote a WebSocket frame decoder in Java:

private byte[] decodeFrame(byte[] _rawIn) {
        int maskIndex = 2;
        byte[] maskBytes = new byte[4];

           


        

Answer 1

Parsing a raw websocket frame is easy enough. But you have to inspect the header one byte at a time.

Here's a rough example:

I left a few TODO's for you to work out on your own (after reading the RFC-6455 spec of course)

Things you can validate:

Base Framing Protocol: RFC-6455 - Section 5.2

• Is the opcode found one of the valid ones defined in the spec?
• Are RSV bits being used improperly?

Client-to-Server Masking: RFC 6455 - Section 5.3

• If Frame was sent by Client, is the Frame Masked?
• Is the Mask random from frame to frame?
• Don't allow [0x00, 0x00, 0x00, 0x00] as a mask.

Fragmentation: RFC 6455 - Section 5.4

• Is it a fragmented Control frame?
• Is the fragmentation of a large message, consisting of multiple frames, out of order?
• Was a new message started before prior one completed with a FIN flag?

Control Frames: RFC 6455 - Section 5.5

• Does the payload length of a control frame exceed 125 bytes?
• Is the payload fragmented?

Close Frames: RFC 6455 - Section 5.5.1

• If a status code is provided in the payload, does the status code conform to one of the status codes declared in section 7.4.1? Don't forget to to check the IANA registry of websocket status codes that were added after the RFC was finalized)
• Is the status code one that is allowed to be sent over the network in a Frame? (see codes 1005, and 1006 for example)
• If a /reason/ is provided in the frame, does it conform to UTF-8 encoding rules?
• Have you received any frames, of any kind, after a Close frame? (this is a no-no)

Data Frames: RFC 6455 - Section 5.6

• If you receive a TEXT payload data (from TEXT + CONTINUATION frames), does the payload data conform to UTF-8 encoding rules?

While you can validate at the individual frame level, you will find that some of the validations above are validations of state and behavior between multiple frames. You can find more of these kinds of validations in Sending and Receiving Data: RFC 6455 - Section 6.

However, if you have extensions in the mix, then you will also need to process the frames from the point of view of the negotiated extension stack as well. Some tests above would appear to be invalid when an extension is being used.

Example: You have Compression Extension (RFC-7692) (such as permessage-deflate) in use, then the validation of TEXT payload cannot be done with the raw frame off the network, as you must first pass the frame through the extension. Note that the extension can change the fragmentation to suit its needs, which might mess up your validation as well.

package websocket;

import java.nio.ByteBuffer;
import java.nio.charset.Charset;

public class RawParse
{
    public static class Frame
    {
        byte opcode;
        boolean fin;
        byte payload[];
    }

    public static Frame parse(byte raw[])
    {
        // easier to do this via ByteBuffer
        ByteBuffer buf = ByteBuffer.wrap(raw);

        // Fin + RSV + OpCode byte
        Frame frame = new Frame();
        byte b = buf.get();
        frame.fin = ((b & 0x80) != 0);
        boolean rsv1 = ((b & 0x40) != 0);
        boolean rsv2 = ((b & 0x20) != 0);
        boolean rsv3 = ((b & 0x10) != 0);
        frame.opcode = (byte)(b & 0x0F);

        // TODO: add control frame fin validation here
        // TODO: add frame RSV validation here

        // Masked + Payload Length
        b = buf.get();
        boolean masked = ((b & 0x80) != 0);
        int payloadLength = (byte)(0x7F & b);
        int byteCount = 0;
        if (payloadLength == 0x7F)
        {
            // 8 byte extended payload length
            byteCount = 8;
        }
        else if (payloadLength == 0x7E)
        {
            // 2 bytes extended payload length
            byteCount = 2;
        }

        // Decode Payload Length
        while (--byteCount > 0)
        {
            b = buf.get();
            payloadLength |= (b & 0xFF) << (8 * byteCount);
        }

        // TODO: add control frame payload length validation here

        byte maskingKey[] = null;
        if (masked)
        {
            // Masking Key
            maskingKey = new byte[4];
            buf.get(maskingKey,0,4);
        }

        // TODO: add masked + maskingkey validation here

        // Payload itself
        frame.payload = new byte[payloadLength];
        buf.get(frame.payload,0,payloadLength);

        // Demask (if needed)
        if (masked)
        {
            for (int i = 0; i < frame.payload.length; i++)
            {
                frame.payload[i] ^= maskingKey[i % 4];
            }
        }

        return frame;
    }

    public static void main(String[] args)
    {
        Charset UTF8 = Charset.forName("UTF-8");

        Frame closeFrame = parse(hexToByteArray("8800"));
        System.out.printf("closeFrame.opcode = %d%n",closeFrame.opcode);
        System.out.printf("closeFrame.payload.length = %d%n",closeFrame.payload.length);

        // Examples from http://tools.ietf.org/html/rfc6455#section-5.7
        Frame unmaskedTextFrame = parse(hexToByteArray("810548656c6c6f"));
        System.out.printf("unmaskedTextFrame.opcode = %d%n",unmaskedTextFrame.opcode);
        System.out.printf("unmaskedTextFrame.payload.length = %d%n",unmaskedTextFrame.payload.length);
        System.out.printf("unmaskedTextFrame.payload = \"%s\"%n",new String(unmaskedTextFrame.payload,UTF8));

        Frame maskedTextFrame = parse(hexToByteArray("818537fa213d7f9f4d5158"));
        System.out.printf("maskedTextFrame.opcode = %d%n",maskedTextFrame.opcode);
        System.out.printf("maskedTextFrame.payload.length = %d%n",maskedTextFrame.payload.length);
        System.out.printf("maskedTextFrame.payload = \"%s\"%n",new String(maskedTextFrame.payload,UTF8));
    }

    public static byte[] hexToByteArray(String hstr)
    {
        if ((hstr.length() < 0) || ((hstr.length() % 2) != 0))
        {
            throw new IllegalArgumentException(String.format("Invalid string length of <%d>",hstr.length()));
        }

        int size = hstr.length() / 2;
        byte buf[] = new byte[size];
        byte hex;
        int len = hstr.length();

        int idx = (int)Math.floor(((size * 2) - (double)len) / 2);
        for (int i = 0; i < len; i++)
        {
            hex = 0;
            if (i >= 0)
            {
                hex = (byte)(Character.digit(hstr.charAt(i),16) << 4);
            }
            i++;
            hex += (byte)(Character.digit(hstr.charAt(i),16));

            buf[idx] = hex;
            idx++;
        }

        return buf;
    }
}