I used this guide to install the \"scientific stack\" for Python (OSX 10.9.2, brewed Python 2.7.6, IPython 2.0, matplotlib 1.3.1, libpng 1.6.10). Everything was looking good
For the sake of documentation (following on from above comments):
Remove X11
launchctl unload /Library/LaunchAgents/org.macosforge.xquartz.startx.plist
sudo launchctl unload /Library/LaunchDaemons/org.macosforge.xquartz.privileged_startx.plist
sudo rm -rf /opt/X11* /Library/Launch*/org.macosforge.xquartz.* /Applications/Utilities/XQuartz.app /etc/*paths.d/*XQuartz
sudo pkgutil --forget org.macosforge.xquartz.pkg
Then,
if you have matplotlib / python installed, run the following as appropriate:
pip uninstall matplotlib
pip uninstall ipython
Then,
pip install matplotlib
pip install ipython
If necessary, XQuartz can be re-installed from https://www.macupdate.com/app/mac/26593/xquartz.
[Note: This does re-add the /opt/X11/include/libpng15/png.h etc files, but ipython worked fine afterwards.]