Is there a way to digitally sign a x509 certificate or any document using openssl?
Yes, the dgst and rsautl component of OpenSSL can be used to compute a signature given an RSA key pair.
openssl dgst -sha256 data.txt > hash
openssl rsautl -sign -inkey privatekey.pem -keyform PEM -in hash >signature
openssl rsautl -verify -inkey publickey.pem -pubin -keyform PEM -in signature
Update: Capturing Reto's comments from below because this is an important nuance. Presumably if you are going to the trouble to verify, you want to know the signature was produced on the plaintext to which it is attached:
This might sound obvious for some but: Be aware, rsault verify just decrypts the file signature. The output of this call is guaranteed to be produced by the owner of the private key, but beside that nothing else is being checked. So to actually verify the consistency of data.txt you have to regenerate the digest and then compare it against the output of openssl rsautl -verify.
data.txt:openssl dgst -sha256 -verify publickey.pem -signature signature data.txt
For this operation, openssl requires the public key, the signature, and the message.