I need to create an OpenSSL connection where I can directly read/write handshake data. The reason is the handshake data will be transported in a UDP connection (DTLS is not
Try doing this
// Connect the TCP socket (this is client side)
int sock = tcp_connect(host, port);
// Create BIO around the connected TCP socket
BIO *sbio = BIO_new_socket(sock, BIO_NOCLOSE);
SSL_set_bio(ssl,sbio,sbio);
int ret = SSL_connect(ssl);
Think of BIO as a wrap around the socket. For client side, you have to give it a socket that is connected to the desired host and port using connect(...) call. For server side, just use the socket that is returned from accept(...) call.