BIO_do_connect fails, returns negative value

Question

I am trying and using OpenSSL in C (on ubuntu 12.04). Took an example from here.

Everything goes well until BIO_do_connect() which returns a negative value. Probably I did something wrong in calling these two API because bio is passed to BIO_do_connect().

An example of the format to use in the second parameters of these two functions would be appreciated.

BIO_set_conn_ip(bio, &ip);
BIO_set_conn_int_port(bio, &port);

It is really not comfortable that both functions return 1 always (correct, wrong or anything), as stated here.

Here the full code:

int main(void) {
    BIO * bio;
    SSL * ssl;
    SSL_CTX * ctx;

    int p;
    char ip[4];
    int port = 60054;

    /* considered big-endian */
    ip[0] = 0b11000000;
    ip[1] = 0b10100100;
    ip[2] =        0b1;
    ip[3] =        0b1110100;

    char * request =
                    "request";
    char r[1024];

    SSL_library_init();

    /* Set up the library */

    ERR_load_BIO_strings();
    SSL_load_error_strings();
    OpenSSL_add_all_algorithms();

    /* Set up the SSL context */

    ctx = SSL_CTX_new(SSLv23_client_method());

    /* Load the trust store */

    if (!SSL_CTX_load_verify_locations(ctx, "cert.pem", NULL)) {
            fprintf(stderr, "Error loading trust store\n");
            ERR_print_errors_fp(stderr);
            SSL_CTX_free(ctx);
            return 0;
    }

    /* Setup the connection */

    bio = BIO_new_ssl_connect(ctx);

    /* Set the SSL_MODE_AUTO_RETRY flag */

    BIO_get_ssl(bio, &ssl);
    SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);

    /* Create and setup the connection */

    BIO_set_conn_ip(bio, &ip);
    BIO_set_conn_port(bio, &port);

    int ret = BIO_do_connect(bio);
    if (ret <= 0) {
            fprintf(stderr, "Error attempting to connect\n");
            ERR_print_errors_fp(stderr);
            BIO_free_all(bio);
            SSL_CTX_free(ctx);
            return 0;
    }

    /* Check the certificate */

    if (SSL_get_verify_result(ssl) != X509_V_OK) {
            fprintf(stderr, "Certificate verification error: %i\n",
                            SSL_get_verify_result(ssl));
            BIO_free_all(bio);
            SSL_CTX_free(ctx);
            return 0;
    }

    /* Send the request */

    BIO_write(bio, request, strlen(request));

    /* Read in the response */

    for (;;) {
            p = BIO_read(bio, r, 1023);
            if (p <= 0)
                    break;
            r[p] = 0;
            printf("%s", r);
    }

    /* Close the connection and free the context */

    BIO_free_all(bio);
    SSL_CTX_free(ctx);
    return 0;
}

This is the output stderr that I get from ERR_print_errors_fp API:

SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40

and when I try and launch this command:

 openssl s_client -connect [ip]:[port] -debug

I get something like the following (of course there is something sensible instead of ++++):

+++++
------
CONNECTED(00000003)
write to 0x9494140 [0x9494418] (225 bytes => 225 (0xE1))
0000 ++++++
---
Certificate chain
 0 ++++++
---
Server certificate
-----BEGIN CERTIFICATE-----
MI++++++
-----END CERTIFICATE-----
subject=/C++++++
---
No client certificate CA names sent
---
SSL handshake has read 931 bytes and written 210 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: DF77194+++
    Session-ID-ctx: 
    Master-Key: 11D6++++
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1394815215
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)

Answer 1

BIO_set_conn_ip() sets the IP address to ip using binary form, that is four bytes specifying the IP address in big-endian form. You are trying to write the IP in the little-endian form. Change the order of bytes like this:

ip[3] = 0b11000000;
ip[2] = 0b10100100;
ip[1] = 0b1;
ip[0] = 0b1110100;

Also, the IP you are trying to set is 192.164.1.116 which might be wrong if you were going for 192.168.1.116 (mind the 168 vs 164 part).