database-performance

I am designing an application, one aspect of which is that it is supposed to be able to receive massive amounts of data into SQL database. I designed the database stricture as a single table with bigint identity, something like this one: CREATE TABLE MainTable ( _id bigint IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED, field1, field2, ... ) I will omit how am I intending to perform queries, since it is irrelevant to the question I have. I have written a prototype, which inserts data into this table using SqlBulkCopy. It seemed to work very well in the lab. I was able to insert tens of millions

Closed . This question is opinion-based. It is not currently accepting answers. Learn more . Want to improve this question? Update the question so it can be answered with facts and citations by editing this post . Closed 5 years ago . I'm not going to make a big long rattle on this question about what I've tested and number crunching. I'm more interested in actual up-to-date practice performances. I've read tons of articles already and some of them are pretty skeptical or either very pro to one library . I'm currently testing a bit with gorp , yet I have no clue how to compare the performances

Implementing an Android+Firebase app, which has a many-to-many relationship: User <-> Widget (Widgets can be shared to multiple users). Considerations: List all the Widgets that a User has. A User can only see the Widgets which are shared to him/her. Be able to see all Users to whom a given Widget is shared. A single Widget can be owned/administered by multiple Users with equal rights (modify Widget and change to whom it is shared). Similar to how Google Drive does sharing to specific users. One of the approaches to implement fetching (join-style), would be to go with this advice: https://www

I have a postgres table (lets call this table Events ) with a composite foreign key to another table (lets call this table Logs ). The Events table looks like this: CREATE TABLE Events ( ColPrimary UUID, ColA VARCHAR(50), ColB VARCHAR(50), ColC VARCHAR(50), PRIMARY KEY (ColPrimary), FOREIGN KEY (ColA, ColB, ColC) REFERENCES Logs(ColA, ColB, ColC) ); In this case, I know that I can efficiently search for Events by the primary key, and join to Logs. What I am interested in is if this foreign key creates an index on the Events table which can be useful even without joining. For example, would the

I'm facing very strange problem, I've asked the question here about speed up the insertion in MYSql, especially about the insertion of Huge SQL files multiple GB in size. They suggested me to use MyISAM engine. I did the following: ALTER TABLE revision ENGINE=MyISAM; Use ALTER TABLE .. DISABLE KEYS . (MyISAM only) Set bulk_insert_buffer_size to 500M. (MyISAM only) Set unique_checks = 0 . not checked. SET autocommit=0; ... SQL import statements ... COMMIT; SET foreign_key_checks=0; It Speed up the process to 5 minutes that previously took 2 hours and I'm impressed. But now when i tried the same

I'm trying to create a Wikipedia DB copy (Around 50GB), but having problems with the largest SQL files. I've split the files of size in GB using linux split utility into chunks of 300 MB. e.g. split -d -l 50 ../enwiki-20070908-page page.input. On average 300MB files take 3 hours at my server. I've ubuntu 12.04 server OS and Mysql 5.5 Server. I'm trying like following: mysql -u username -ppassword database < category.sql Note: these files consist of Insert statements and these are not CSV files. Wikipedia offers database dumps for download, so everybody can create a copy of Wikipedia. You can