I have done all kinds of research and tried many different things. I know this question has been answered many times, but none of the suggested solutions are working for me.
I actually had 2 versions of libxml installed, one from source, one from an RPM.
The following is my complete solution
I uninstalled source (from the source directory)
sudo make uninstall
Remove bundles
rm -rf ~/.bundle ~/.bundler
Updated LD (might have to do this as root, not sudo)
sudo ldconfig
Then reinstalled the bundle
bundle install