How to create a fast & huge union array without wasting memory in C?

问题

I'd like to store different datatypes in the same memory using union. This array has a fixed length and shall be accessed quickly and shall waste as little memory as possible.

I will define areas in which the same datatypes are stored. So I do this:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <inttypes.h>

#define RESERVED_BYTES 1073741824
//#define RESERVED_BYTES 2147483648

typedef union {
    char c[RESERVED_BYTES];             //1Byte
    uint8_t u8[RESERVED_BYTES];         //1Byte
    uint16_t u16[RESERVED_BYTES / 2];   //2Byte
} array_type;

int main(void)
{
    array_type *array;
    array = calloc(1, sizeof(array_type));

    getchar();
    return 0;
}

This code works and allocates 1GB of memory and the elements of that array can be used with array[0].u8[3] e.g. I know I have to take care of indices because they depend on the byte-size.

Sadly the code fails to compile if I increase the memory size(see #define RESERVED_BYTES 2147483648) and with MSVS 2013 on a 64bit machine I receive an error C2148: total size of array must not exceed 0x7fffffff bytes. On the other hand it is no problem to put the 2GB straight into the calloc like that array = calloc(2147483648, sizeof(*array));.

But with this version I might waste memory:

union {
    char c;             //1Byte
    uint8_t u8;         //1Byte
    uint16_t u16;   //2Byte
} *array;

or need to build a time-consuming function that calculates the both indices I want to avoid:

union {
    char c[2];             //1Byte
    uint8_t u8[2];          //1Byte
    uint16_t u16[1];    //2Byte
} *array;
array[3].u8[2] = 1;

So how to deal with that issue?

回答1:

I'd like to store different datatypes in the same memory using union. This array has a fixed length

Going by the code you posted, the arrays have the same byte-length, but hold different number of elements based on their type.

I receive an error C2148: total size of array must not exceed 0x7fffffff bytes.

If I were to guess, this might be related to the 2 GB limit on static data that applies even to 64-bit compiles, meaning that a union of type array_type could never be instantiated as either a global/static variable, or a local/stack one (Memory Limits for Applications on Windows). Which, in the end, means that the only choice left is to allocate such an array dynamically on the heap.

But with this version I might waste memory
...or need to build a time-consuming function that calculates the both indices

You could achieve (nearly) the same effect by slightly modifying the union definition, without memory waste and without additional accessors.

#define RESERVED_BYTES 2147483648

typedef union {
    void *pv;
    char *pc;
    uint8_t *pu8;
    uint16_t *pu16;
} parray_type;

int main(void)
{
    parray_type parray;
    parray.pv = calloc(RESERVED_BYTES, 1);

    // last element in allocated buffer for each type
    char c = parray.pc[RESERVED_BYTES - 1];
    uint8_t u8 = parray.pu8[RESERVED_BYTES - 1];
    uint16_t u16 = parray.pu16[RESERVED_BYTES / 2 - 1];

    return 0;
}

Of course, you'll have to always remember that the max index for pu16 is half that of pc and pu8 but that's the premise in the original code as well.

来源：https://stackoverflow.com/questions/34624027/how-to-create-a-fast-huge-union-array-without-wasting-memory-in-c

标签

arrays

memory

unions