batch-insert

How to commit batches of inserts in hibernate?

一曲冷凌霜 提交于 2019-12-10 15:19:29
问题 I have to save a large number of objects to the database using hibernate. Instead of commiting all of them at once, I would like to commit as soon as n (BATCH_SIZE) objects are present in the session. Session session = getSession(); session.setCacheMode(CacheMode.IGNORE); for(int i=0;i<objects.length;i++){ session.save(objects[i]); if( (i+1) % BATCH_SIZE == 0){ session.flush(); session.clear(); } } I would have tried something like above, but I read that session.flush() does not commit the

JDBC Batch executing extremely slow

末鹿安然 提交于 2019-12-07 01:42:36
问题 Can anyone tell me what I'm doing wrong I'm executing 350 inserts in a mysql and it's taking like 40 secs. Here is the code long t0 = System.currentTimeMillis(); Connection con = connectionProvider.getConnection(); PreparedStatement s = con.prepareStatement("insert into domkee.friends(idFriends,friend1Id,friend2Id,friend2Name) values(?,?,?,?)"); con.setAutoCommit(false); for (Friend f : friends) { s.setLong(1, 0); s.setLong(2, f.getFriend1Id()); s.setLong(3, f.getFriend2Id()); s.setString(4,

Inserting large number of records without locking the table

一笑奈何 提交于 2019-12-06 01:52:32
问题 I am trying to insert 1,500,000 records into a table. Am facing table lock issues during the insertion. So I came up with the below batch insert. DECLARE @BatchSize INT = 50000 WHILE 1 = 1 BEGIN INSERT INTO [dbo].[Destination] (proj_details_sid, period_sid, sales, units) SELECT TOP(@BatchSize) s.proj_details_sid, s.period_sid, s.sales, s.units FROM [dbo].[SOURCE] s WHERE NOT EXISTS (SELECT 1 FROM dbo.Destination d WHERE d.proj_details_sid = s.proj_details_sid AND d.period_sid = s.period_sid)

JDBC Batch executing extremely slow

血红的双手。 提交于 2019-12-05 05:42:08
Can anyone tell me what I'm doing wrong I'm executing 350 inserts in a mysql and it's taking like 40 secs. Here is the code long t0 = System.currentTimeMillis(); Connection con = connectionProvider.getConnection(); PreparedStatement s = con.prepareStatement("insert into domkee.friends(idFriends,friend1Id,friend2Id,friend2Name) values(?,?,?,?)"); con.setAutoCommit(false); for (Friend f : friends) { s.setLong(1, 0); s.setLong(2, f.getFriend1Id()); s.setLong(3, f.getFriend2Id()); s.setString(4, f.getFriend2Name()); s.addBatch(); } long t1 = System.currentTimeMillis() - t0; s.executeBatch(); long

Inserting large number of records without locking the table

时间秒杀一切 提交于 2019-12-04 05:43:37
I am trying to insert 1,500,000 records into a table. Am facing table lock issues during the insertion. So I came up with the below batch insert. DECLARE @BatchSize INT = 50000 WHILE 1 = 1 BEGIN INSERT INTO [dbo].[Destination] (proj_details_sid, period_sid, sales, units) SELECT TOP(@BatchSize) s.proj_details_sid, s.period_sid, s.sales, s.units FROM [dbo].[SOURCE] s WHERE NOT EXISTS (SELECT 1 FROM dbo.Destination d WHERE d.proj_details_sid = s.proj_details_sid AND d.period_sid = s.period_sid) IF @@ROWCOUNT < @BatchSize BREAK END I have a clustered Index on Destination table (proj_details_sid

How to multi insert rows in cassandra

房东的猫 提交于 2019-12-04 01:38:08
What is the most efficient way of inserting multiple rows in cassandra column family. Is it possible to do this in a single call. Right now my approach is to addinsert multiple column and then execute. There in a single call I am persisting one row. I am looking for strategy so that I can do a batch insert. Priyank Desai CQL contains a BEGIN BATCH...APPLY BATCH statement that allows you to group multiple inserts so that a developer can create and execute a series of requests (see http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0 ). The following worked for me (Scala):

How to insert records using select in codeigniter active record

扶醉桌前 提交于 2019-12-03 08:25:22
I want to implement a sql query using CodeIgniter Active Record class. The query looks like this.. INSERT california_authors (au_id, au_lname, au_fname) SELECT au_id, au_lname, au_fname FROM authors WHERE State = 'CA' Is this possible in CodeIgniter without using the $this->db->query method? Solution : $this->db->select('au_id, au_lname, au_fname'); $this->db->from('california_authors'); $this->db->where('state', 'CA'); $query = $this->db->get(); if($query->num_rows()) { $new_author = $query->result_array(); foreach ($new_author as $row => $author) { $this->db->insert("authors", $author); } }

Batch insert in Laravel 5.2

白昼怎懂夜的黑 提交于 2019-12-01 15:41:53
I am using a API's with lot's of calculation almost 100 database fields at the end with a big Foreach loop. In every iteration i insert data in database. I want to insert data in once at the end (Batch Insert like in CodeIgniter). Any body have idea how to insert all data at the end of iteration. instead of every iteration it insert row in database. I want to insert data at the end of loop. Any help or idea appreciated. Use insert() method for bulk insertion. First, build an array with this structure: $data = [ ['name' => 'John', 'age' => 25], ['name' => 'Maria', 'age' => 31], ['name' =>

Batch insert in Laravel 5.2

梦想与她 提交于 2019-12-01 14:24:50
问题 I am using a API's with lot's of calculation almost 100 database fields at the end with a big Foreach loop. In every iteration i insert data in database. I want to insert data in once at the end (Batch Insert like in CodeIgniter). Any body have idea how to insert all data at the end of iteration. instead of every iteration it insert row in database. I want to insert data at the end of loop. Any help or idea appreciated. 回答1: Use insert() method for bulk insertion. First, build an array with

How to perform batch update in Spring with a list of maps?

老子叫甜甜 提交于 2019-11-30 17:39:05
New to Spring, I am trying to insert a List<Map<String, Object>> into a table. Until now I have been using the SqlParameterSource for batch update, which works fine when a java bean is supplied to them. Something like this: @Autowired private NamedParameterJDBCTemplate v2_template; public int[] bulkInsertIntoSiteTable(List<SiteBean> list){ SqlParameterSource[] batch = SqlParameterSourceUtils .createBatch(list.toArray()); int[] updateCounts = v2_template .batchUpdate( "insert into sitestatus (website, status, createdby) values (:website, :status, :username)", batch); return updateCounts; }