How to get past 1gb memory limit of 64 bit LuaJIT on Linux?

Question

The overview is I am prototyping code to understand my problem space, and I am running into \'PANIC: unprotected error in call to Lua API (not enough memory)\' errors. I am

Answer 1

Here is some follow-up information for those who find this question later:

The key information is as posted by Colonel Thirty Two, that C module extensions and FFI code can easily get outside of the limit. (and the referenced lua list post reminds that plain Lua tables that go outside the limit will be very slow to garbage collect)

It took me some time to pull the pieces together to both access and save/load my objects, so here it is in one place:

I used lds at https://github.com/neomantra/lds as a starting point, in particular the 1-D Array code.

This broke using torch.save(), as it doesn't know how to write the new objects. For each object I added the code below (using Array as the example):

function Array:load(inp)
   for i=1,#inp do
      self._data[i-1] = tonumber(inp[i])
   end
   return self
end

function Array:serialize ()
   local siz = tonumber(self._size)
   io.write(' lds.ArrayT( ffi.typeof("double"), lds.MallocAllocator )( ', siz , "):load({")
   for i=0,siz-1 do
      io.write(string.format("%a,", self._data[i]))
   end
   io.write("})")
end

Note that my application specifically uses doubles and malloc(), so a better implementation would store and use these in self rather than hard coding above.

Then as discussed in PiL and elsewhere, I needed a serializer that would handle the object:

function serialize (o)
     if type(o) == "number" then
       io.write(o)
     elseif type(o) == "string" then
       io.write(string.format("%q", o))
     elseif type(o) == "table" then
       io.write("{\n")
       for k,v in pairs(o) do
          io.write("  ["); serialize(k); io.write("] = ")
         serialize(v)
         io.write(",\n")
       end
       io.write("}\n")
     elseif o.serialize then
        o:serialize()
     else
       error("cannot serialize a " .. type(o))
     end
end

and this needs to be wrapped with:

io.write('do local _ = ')
serialize( myWeirdTable )
io.write('; return _; end')

and then the output from that can be loaded back in with

local myWeirdTableReloaded = dofile('myWeirdTableSaveFile')

See PiL (Programming in Lua book) for dofile()

Hope that helps someone!

Answer 2

You can use the torch tds module. From the README:

Data structures which do not rely on Lua memory allocator, nor being limited by Lua garbage collector.

Only C types can be stored: supported types are currently number, strings, the data structures themselves (see nesting: e.g. it is possible to have a Hash containing a Hash or a Vec), and torch tensors and storages. All data structures can store heterogeneous objects, and support torch serialization.

Answer 3

Only object allocated by LuaJIT itself are limited to the first 2GB of memory. This means that tables, strings, full userdata (i.e. not lightuserdata), and FFI objects allocated with ffi.new will count towards the limit, but objects allocated with malloc, mmap, etc. are not subjected to this limit (regardless if called by a C module or the FFI).

An example for allocating a structure with malloc:

ffi.cdef[[
    typedef struct { int bar; } foo;
    void* malloc(size_t);
    void free(void*);
]]

local foo_t = ffi.typeof("foo")
local foo_p = ffi.typeof("foo*")

function alloc_foo()
    local obj = ffi.C.malloc(ffi.sizeof(foo_t))
    return ffi.cast(foo_p, obj)
end

function free_foo(obj)
    ffi.C.free(obj)
end

The new GC to be implemented in LuaJIT 3.0 IIRC will not have this limit, but I haven't heard any news on it's development recently.

Source: http://lua-users.org/lists/lua-l/2012-04/msg00729.html