Both connect() and bind() system calls \'associate\' the socket file descriptor to an address (typically an ip/port combination). Their prototypes
To make understanding better , lets find out where exactly bind and connect comes into picture,

Further to positioning of two calls , as clarified by Sourav,
bind() associates the socket with its local address [that's why server side binds, so that clients can use that address to connect to server.] connect() is used to connect to a remote [server] address, that's why is client side, connect [read as: connect to server] is used.
We cannot use them interchangeably (even when we have client/server on same machine) because of specific roles and corresponding implementation.
I will further recommend to correlate these calls TCP/IP handshake .

So , who will send SYN here , it will be connect() . While bind() is used for defining the communication end point.
Hope this helps!!