When you have code like:
Bitmap bmp = new Bitmap ( 100, 100 );
Graphics g = Graphics.FromImage ( bmp );
Pen p = new Pen ( Color.FromArgb ( 128, Color.Blue )
Yes, bmp, g, b and p are all IDisposable, you should Dispose() all of them. Preferably by using using() {} blocks.
There are exceptions, when you use Pen p2 = Pens.Blue; yous should not dispose p2. It's called a stock item. The same for Brushes.Black etc.
As for the why, it's the same for all disposable classes. .Net does not use reference counting so there is no (cannot be) an immediate action when a reference goes out of scope.
And leaving it to the Garbage collector will eventually free them but it is (very) inefficient. I know of an ASP.NET (!) application that failed on a shortage of Graphic handles because of not promptly disposing them. It was generating images.