Neo4j get pretty slower with high transaction concurrency

问题

I was using neo4j 3.1.0 enterprise edition. The main logical in my graph is: There are "IP" nodes and "User" nodes and both have "UNIQUE" constraints. Each time an user login, I add a relationship from IP to User.

Here is my insert Cypher:

MERGE (i:IP {ip:"1.2.3.4"}) 
MERGE (u:User {username:"xxx@gmail.com"}) 
MERGE (i) - [l:SUCC] -> (u) 
SET i:ExpireNode, i.expire={expire} 
SET u:ExpireNode, u.expire={expire}
SET l.expire={expire}, l.login={login}

The insert is pretty fast as the beginning. But when the number of node grows to millions, it became very slow and sometimes took more than 1 second to insert nodes and relationships.

How could I optimize it? I was running neo4j with 12-cores CPU and 64G memory. The initial head size is 16G and page cache is 30G.

--------------------------------------------------------------

Tested the same cypher in Web UI and it took 10ms for commands. But by using java driver, it will sometimes take more than 1s. Below is my java code:

try (Transaction tx = session.beginTransaction()) {
    for (Login login : loginList) {
        Value value = login2Operation(login);
        tx.run(INSERT_COMMANDS_SUCC, value);
    }
    tx.success();
}

--------------------------------------------------------------

After some exploring, I found that the insert speed increased significantly if running in 5 threads. But the overall speed is too slow and I have to increase to 100 threads. Then the single insert of each speed grows to 1s. So, I believe the problem is because the parallel ability of Neo4j.

In the neo4j.conf, I added dbms.threads.worker_count=200. But it's not helping. Any ideas?

回答1:

Thanks to @InverseFalcon 's advices and the UNWIND operation helps a lot!

Get more details in Michael Hunger's tips and tricks

来源：https://stackoverflow.com/questions/42507239/neo4j-get-pretty-slower-with-high-transaction-concurrency