If you used rbenv to install it, you can use
rbenv versions
to see which versions you have installed.
Then, use the uninstall command:
rbenv uninstall [-f|--force]
for example:
rbenv uninstall 2.4.0 # Uninstall Ruby 2.4.0
If you installed Rails, it will be removed, too.