I always run into surprises when using the IOStream.
The library seems text oriented and not binary oriented. That may be the first surprise: using the binary flag in file streams is not sufficient to get binary behavior. User Charles Salvia above has observed it correctly: IOStreams mixes formatting aspects (where you want pretty output, e.g. limited digits for floats) with serialization aspects (where you do not want information loss). Probably it would be good to separate these aspects. Boost.Serialization does this half. You have a serialize function which routes to the inserters and extractors if you want. There already you have the tension between both aspects.
Many functions have also confusing semantics (e.g. get, getline, ignore and read. Some extract the delimiter, some don't; also some set eof). Further on some mention the weird function names when implementing a stream (e.g. xsputn, uflow, underflow). Things get even worse when one uses the wchar_t variants. The wifstream does a translation to multibyte while wstringstream does not. Binary I/O does not work out of the box with wchar_t: you have the overwrite the codecvt.
The c buffered I/O (i.e. FILE) is not as powerful as its C++ counterpart, but is more transparent and has much less counter intuitive behavior.
Still every time when I stumble upon the IOStream, I get attracted to it like a moth to fire. Probably it would be a good thing if some really clever guy would have a good look at the overall architecture.