Java array with more than 4gb elements

Question

I have a big file, it\'s expected to be around 12 GB. I want to load it all into memory on a beefy 64-bit machine with 16 GB RAM, but I think Java does not support byte arra

Answer 1

don't limit your self with Integer.MAX_VALUE

although this question has been asked many years ago, but a i wanted to participate with a simple example using only java se without any external libraries

at first let's say it's theoretically impossible but practically possible

a new look : if the array is an object of elements what about having an object that is array of arrays

here's the example

import java.lang.reflect.Array;
import java.util.ArrayList;
import java.util.List;

/**
*
* @author Anosa
*/
 public class BigArray{

private final static int ARRAY_LENGTH = 1000000;

public final long length;
private List arrays;

public BigArray(long length, Class glasss)
{
    this.length = length;
    arrays = new ArrayList<>();
    setupInnerArrays(glasss);

}

private void setupInnerArrays(Class glasss)
{
    long numberOfArrays = length / ARRAY_LENGTH;
    long remender = length % ARRAY_LENGTH;
    /*
        we can use java 8 lambdas and streams:
        LongStream.range(0, numberOfArrays).
                        forEach(i ->
                        {
                            arrays.add((t[]) Array.newInstance(glasss, ARRAY_LENGTH));
                        });
     */

    for (int i = 0; i < numberOfArrays; i++)
    {
        arrays.add((t[]) Array.newInstance(glasss, ARRAY_LENGTH));
    }
    if (remender > 0)
    {
        //the remainer will 100% be less than the [ARRAY_LENGTH which is int ] so
        //no worries of casting (:
        arrays.add((t[]) Array.newInstance(glasss, (int) remender));
    }
}

public void put(t value, long index)
{
    if (index >= length || index < 0)
    {
        throw new IndexOutOfBoundsException("out of the reange of the array, your index must be in this range [0, " + length + "]");
    }
    int indexOfArray = (int) (index / ARRAY_LENGTH);
    int indexInArray = (int) (index - (indexOfArray * ARRAY_LENGTH));
    arrays.get(indexOfArray)[indexInArray] = value;

}

public t get(long index)
{
    if (index >= length || index < 0)
    {
        throw new IndexOutOfBoundsException("out of the reange of the array, your index must be in this range [0, " + length + "]");
    }
    int indexOfArray = (int) (index / ARRAY_LENGTH);
    int indexInArray = (int) (index - (indexOfArray * ARRAY_LENGTH));
    return arrays.get(indexOfArray)[indexInArray];
}

}

and here's the test

public static void main(String[] args)
{
    long length = 60085147514l;
    BigArray array = new BigArray<>(length, String.class);
    array.put("peace be upon you", 1);
    array.put("yes it worj", 1755);
    String text = array.get(1755);
    System.out.println(text + "  i am a string comming from an array ");

}

this code is only limited by only Long.MAX_VALUE and Java heap but you can exceed it as you want (I made it 3800 MB)

i hope this is useful and provide a simple answer