Here is my piece of (a former OpenGL toyer's) advice on the choice of libraries and tools (as for the compiler, you can just use any free modern one out there (i.e., gcc or VSExpress C++)):
Do not forget to look at GLEW (The OpenGL Extension Wrangler Library) or GLee (GL Easy Extension library), especially if you'd like to be stick with more modern OpenGL features (including advanced shaders).
Something named like Shader Debugger will probably also be of great benefit when toying with shaders, as it will give the opportunity to see shader effect immediately without even compiling any C/C++ sources.
Looking at the "traditional" (covered in the (free now) "Red book") functionality from the modern (i.e. shaders again:)) viewpoint may also be enlightening, the "Orange book" ("OpenGL Shading Language (3rd Edition))" has a chapter on emulating OpenGL fixed functionality with shaders.
