Setting smartindent on makes Vim behave like you describe for me, whereas with nosmartindent (which is what I tend to use) it behaves like you'd prefer it to.
Update: From the docs on smartindent:
  When typing '#' as the first character in a new line, the indent for
  that line is removed, the '#' is put in the first column.  The indent
  is restored for the next line.  If you don't want this, use this
  mapping: ":inoremap # X^H#", where ^H is entered with CTRL-V CTRL-H.
  When using the ">>" command, lines starting with '#' are not shifted
  right.
That seems to be it.
Update: Probably no need to bother with the following... I'll leave it here for the added informational value. ;-)
If setting nosmartindent doesn't help, perhaps you could use the :set command -- with no parameters -- to obtain the list of all settings in effect in your Vim session, then paste it somewhere (on Pastie perhaps). There's a few other options which affect automatic indentation, as far as I remember.