bulkinsert

Is there a more efficient way for doing this? for item in item_list: e, new = Entry.objects.get_or_create( field1 = item.field1, field2 = item.field2, ) You can't do decent bulk insertions with get_or_create (or even create), and there's no API for doing this easily. If your table is simple enough that creating rows with raw SQL isn't too much of a pain, it's not too hard; something like: INSERT INTO site_entry (field1, field2) ( SELECT i.field1, i.field2 FROM (VALUES %s) AS i(field1, field2) LEFT JOIN site_entry as existing ON (existing.field1 = i.field1 AND existing.field2 = i.field2) WHERE

I am trying to insert a .csv file into a database with unix linebreaks. The command I am running is: BULK INSERT table_name FROM 'C:\file.csv' WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '\n' ) If I convert the file into Windows format the load works, but I don't want to do this extra step if it can be avoided. Any ideas? I felt compelled to contribute as I was having the same issue, and I need to read 2 UNIX files from SAP at least a couple of times a day. Therefore, instead of using unix2dos, I needed something with less manual intervention and more automatic via programming. As noted, the

In my table I insert around 20,000 rows on each load. Right now I am doing it one-by-one. From mysql website I came to know inserting multiple rows with single insert query is faster. Can I insert all 20000 in single query? What will happen if there are errors within this 20000 rows? how will mysql handle that? If you are inserting the rows from some other table then you can use the INSERT ... SELECT pattern to insert the rows. However if you are inserting the values using INSERT ... VALUES pattern then you have the limit of max_allowed_packet. Also from the docs:- To optimize insert speed,

How can we load data to Mysql Tables from XML Files?? Is there any way to read data from XML Files and Write to MySql database.. I have a bulk of data in XML Files. Thanks in Advance for help. Try the LOAD XML function (MySQL 6.0). Here's the sample code from the reference manual: Using an XML document person.xml containing: <?xml version="1.0"?> <list> <person person_id="1" fname="Pekka" lname="Nousiainen"/> <person person_id="2" fname="Jonas" lname="Oreland"/> <person person_id="3"><fname>Mikael</fname><lname>Ronström</lname></person> <person person_id="4"><fname>Lars</fname><lname>Thalmann<

Possible Duplicate: Any way to SQLBulkCopy “insert or update if exists”? I am using SQLBulkCopy to insert Bulk records How can I perform on update (rather than an insert) on records that already exist? Is this possible with SQLBulkCopy ? This is my code for SQLBulkCopy using (var bulkCopy = new SqlBulkCopy(ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString, SqlBulkCopyOptions.KeepNulls & SqlBulkCopyOptions.KeepIdentity)) { bulkCopy.BatchSize = CustomerList.Count; bulkCopy.DestinationTableName = "dbo.tCustomers"; bulkCopy.ColumnMappings.Clear(); bulkCopy.ColumnMappings