Implementing Queue for data to be inserted in the database

问题

We are developing a vehicle tracking system in which several GPS devices keep sending their GPS locations to the server using TCP connection. The TCP communicator decodes the GPS location and inserts that data into the database. Right now, one thread of TCP communicator serves one device request. After decoding the GPS data, it creates a connection to the database and inserts the data and releases the connection. As number of devices are increasing, the number of concurrent connections to the database (which is MySQL) are also increasing. Hence, I want to implement a queue where each thread of TCP communicator will push the data to one end and one job will take data from other end and keep it inserting into the database.

Can anybody suggest me the best solution to handle all this? Our application is based on Java and database is MySQL.

Thanks, Saurabh

回答1:

You can use a thread-safe queue implementation like ConcurrentLinkedQueue to queue the data

回答2:

You could just create a simple Thread that handles the database writes. Then have your communicator threads queue the data that needs to be written with it. Something like this:

public class DatabaseQueue extends Thread {
    private LinkedBlockingQueue<Data> queue = new LinkedBlockingQueue<Data>();

    public void queueData(Data data) {
        queue.add(data);
    }

    public void run() {
        while (true) {
            Data data = queue.take();
            // write data to database
        }
    }
}

回答3:

A plain concurrent queue is best if you are going to batch your data and it support easy batching techniques (and therefor database insert performance)

However a more flexible approach if you want to do other things as well is to use an ExecutorService with a fixed numebr of threads. This way you can add tasks to do anything, to a limited degree of concurrency.

回答4:

You need to implement a message queue.

Have a look at either RabbitMQ or ActiveMQ.

回答5:

one job will take data from other end and keep it inserting into the database.

ConcurrentLinkedQueue is just an overkill, it is for more than one thread to access lock free.The poll method take unnecessary sleep intervals,also u need to wait till the queue is filled , again another kind of sleep. If you only have one thread putting stuff into the queue, and another thread taking stuff out of the queue

Queue<GPRSObj> queue = Collections.synchronizedList(new LinkedList<GPRSObj>());

LinkedBlockingQueue

来源：https://stackoverflow.com/questions/5499350/implementing-queue-for-data-to-be-inserted-in-the-database

标签

java

mysql

queue