If you need to worry about compatibility between different versions of the server and client, I'd use Google protocol buffers or JSON (either an existing implementation, or write your own). If the versions will always be in sync, just write your own binary serialization, or even tailor the in-memory structure to be a 'serialized' form and use access functions to read and write it. This basically entails storing numbers as a fixed number of bytes in a fixed-endian order, or some vlc format if you want to get fancy, and using object handles of some sort (simplest is an offset into an array) in place of explicit pointers when one object needs to refer to another.
One really nice side benefit of keeping data in a 'serialized' form internally is that you can work with huge data sets on disk (for example via mmap) without having to write your own complex caching code.
