How to add objects to a distributed queue in hazelcast performantly?

Question

Adding items to a distributed queue in hazelcast is incredibly slow (read: 66 items/sec; is that normal?) when the queue is stored on a different node than the one where the code is executed ( and is set to 0 in the config for this queue). Is there any way to add the items from the owner node? Is there anything fundamentally wrong with my approach using Hazelcast?

This operation takes around 15 seconds:

public static void main(String[] args) throws ExecutionException {
    HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
    IQueue<String> configs = hazelcastInstance.getQueue("configs");  
    for(int i = 0; i<1000;i++) {
        configs.add("Some string"+i);
    }
}

Changing the values and in the config (see below) doesn't have any influence on the execution speed. I'd assume that increasing would block the insert operations and increasing would not (actually the loop should be run through as quickly as if the #add operation was on a local queue). However, the time executing the for-loop is the same. Even if i set both values to 0. Am I missing something?

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xsi:schemaLocation=
  "http://www.hazelcast.com/schema/config hazelcast-config-3.7.xsd"
  xmlns="http://www.hazelcast.com/schema/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <network>
        <port auto-increment="true" port-count="20">5701</port>
        <join>
            <multicast enabled="false">
        </multicast>
        <tcp-ip enabled="true">
            <member>172.105.66.xx</member> 
        </tcp-ip>
        </join>
    </network>

    <queue name="configs">
        <statistics-enabled>false</statistics-enabled>
        <max-size>0</max-size>
        <backup-count>0</backup-count>
        <async-backup-count>1</async-backup-count>
        <empty-queue-ttl>-1</empty-queue-ttl>
    </queue>
</hazelcast>

Answer 1

there are couple of ways you can optimize the Queue. The queue can store your entries in a binary or in object form. On top you can define the number of backups, the more backups the slower would it be. The default number of replicated backups is 1. This mean that for every pu you will have one replication besides the put. Probably the thing that increases the performance the most when using binary form of entry storage is the serialization. Java serialization is very slow. Use DataSerializeable or IdentifiedDataserializeable instead.

You need aso to check your network latency for some slow networks you would have 1-2MS latency only for network comunication for 1000 inserts this would be 1000 - 2000 ms which is 1-2 second only network waiting.

For batch iperations you can do the inserts in paralel as long as the order of the inserts is not important.

And Queue can have asynchroneus backups which would again incease a bit the performance.

UPDATE: During normal running of the application you will rarely have a single theaded scenario. So instead of inserting these 1000 values one by one do the following:
List arrayList = new ArrayList();

for(int i=0;i<1000;++i) {
   arrayList.add("myString " + i);
}

arrayList.paralelstream().forEach(t->configs.put(t));

Now you are inserting in paralel .