Java Neo4J out of memory

问题

This is a bit like this: Neo4j OutOfMemory problem

But it's outdated and apparently so are the solutions as far as I can tell.

So I'm trying to insert around 100K nodes with 5.5M relations (I actually cut down my data set so it's now more like <100K nodes with 2.8M relations).

After a while, it runs out of memory and I get an exception like so:

Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.StringBuilder.toString(Unknown Source)
    at org.neo4j.kernel.impl.util.StringLogger$ActualStringLogger.logMessage(StringLogger.java:276)
    at org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:85)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.LinkedList.addBefore(Unknown Source)
    at java.util.LinkedList.add(Unknown Source)
    at org.neo4j.kernel.impl.nioneo.store.IdGeneratorImpl.freeId(IdGeneratorImpl.java:291)
    at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.freeId(CommonAbstractStore.java:382)
    at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doRollback(WriteTransaction.java:315)
    at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.rollback(XaTransaction.java:278)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:518)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:558)
    at org.neo4j.kernel.impl.transaction.TxManager.rollback(TxManager.java:610)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:129)
    at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:119)
    at sqlToGraph.SqlToGraph.main(SqlToGraph.java:81)

I've tried passing -Xmx1500m to java, which is about the limit of what I can pass because before it complains about not being able to allocate the heap space. It lasts significantly longer, but still doesn't finish.

Here is the (slightly edited) code:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    while (rs.next()) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();

回答1:

Adding another answer here. So given the code the problem is that you never commit your transaction. Transaction data is kept in memory until committed so all your created nodes and relationships will just sit in memory awaiting a commit and that's why you eventually get OOM.

I'd propose this code change:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    for (int i = 0; rs.next(); i++) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));

        // Commit every now and then to free memory.
        if ( i > 0 && i % 10000 == 0 ) {
            tx.success();
            tx.finish();
            tx = graphDb.beginTx();
        }
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();

回答2:

If splitting up the transaction is not feasible for you as you rely on the atomicity guarantee of a single transaction, another option might be to simplify the data model, e.g. reduce the number of properties added to nodes/relationships.

In our case we could fix an OOM issue by omitting properties that held default values or zeros.

来源：https://stackoverflow.com/questions/11103436/java-neo4j-out-of-memory