Imagine you have the following array of integers:
array(1, 2, 1, 0, 0, 1, 2, 4, 3, 2, [...] );
The integers go on up to one million entries; on
APC stores the data serialized, so it has to be unserialized as it is loaded back from APC. That's where your overhead is.
The most efficient way of loading it is to write to file as PHP and include(), but you're never going to have any level of efficiency with an array containing a million elements... it takes an enormous amount of memory, and it takes time to load. This is why databases were invented, so what is your problem with a database?
EDIT
If you want to speed up serialize/deserialize, take a look at the igbinary extension