database-performance

I use Talend to load data into a sql-server database. It appears that the weakest point of my job is not the dataprocessing, but the effective load in my database, which is not faster than 17 rows/sec. The funny point is that I can launch 5 jobs in the same time, and they'll all load at 17rows/sec . What could explain this slowness and how could I improve the speed? Thanks New informations: The transfer speed between my desktop and the server is about 1MByte My job commits every 10 000 I use sql server 2008 R2 And the schema I use for my jobs is like this: Database INSERT OR UPDATE methods are

I have written a daemon processor which will fetch the records from one database and insert them into another database for synchronizing. It will fetch records based on each record indication flag which is boolean datatype. My tables has hundreds of thousands of records. When I select the record whichever sync_done is false, will it cause any database performance issues? Or should I apply indexing for that sync_done column (boolean datatype), to improve performance, since it will apply select operation on records with a sync_done value of false? For example, say I have 10000 records. Of those,

We have an InnoDB database that is about 70 GB and we expect it to grow to several hundred GB in the next 2 to 3 years. About 60 % of the data belong to a single table. Currently the database is working quite well as we have a server with 64 GB of RAM, so almost the whole database fits into memory, but we’re concerned about the future when the amount of data will be considerably larger. Right now we’re considering some way of splitting up the tables (especially the one that accounts for the biggest part of the data) and I’m now wondering, what would be the best way to do it. The options I’m

I have this process that has to make a series of queries, using pl/pgsql: --process: SELECT function1(); SELECT function2(); SELECT function3(); SELECT function4(); To be able to execute everything in one call, I created a process function as such: CREATE OR REPLACE FUNCTION process() RETURNS text AS $BODY$ BEGIN PERFORM function1(); PERFORM function2(); PERFORM function3(); PERFORM function4(); RETURN 'process ended'; END; $BODY$ LANGUAGE plpgsql The problem is, when I sum the time that each function takes by itself, the total is 200 seconds, while the time that the function process() takes

i want to store some data in my neo4j database. i use spring-data-neo4j for that. my code is like the follow: for (int i = 0; i < newRisks.size(); i++) { myRepository.save(newRisks.get(i)); System.out.println("saved " + newRisks.get(i).name); } My newRisks-array contains circa 60000 objects and 60000 edges. Every node and edge has one property. The duration of this loop is circa 15 - 20 minutes, is this normal? I used Java VisualVM to search some bottlenecks, but my average CPU usage was 10 - 25% (of 4 cores) and my heap was less than half full. There are any options to boost up this operation

I have a simple mysql query, but when I have a lot of records (currently 103,0000), the performance is really slow and it says it is using filesort, im not sure if this is why it is slow. Has anyone any suggestions to speed it up? or stop it using filesort? MYSQL query : SELECT adverts .* FROM adverts WHERE ( price >='0' ) AND ( adverts.status = 1 ) AND ( adverts.approved = 1 ) ORDER BY date_updated DESC LIMIT 19990 , 10 The Explain results : id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE adverts range price price 4 NULL 103854 Using where; Using filesort Here is