Bulk insert of hundreds of millions of records

后端 未结 3 1276
梦毁少年i
梦毁少年i 2020-12-30 17:29

What is the fastest way to insert 237 million records into a table that has rules (for distributing data across child tables)?

I have tried or considered:

3条回答
  •  南方客
    南方客 (楼主)
    2020-12-30 17:35

    1. create parent table without any index, only column and there types (create table some_data (c_1 int, c_2 varchar,....))
    2. create sequence for new data tables enumeration
    3. take new id from sequence
    4. create new table for real data with 'like' key word (create table some_data_X like some_data)
    5. insert real data in some_data_X with copy in binary format
    6. create indexes, time constraints etc (empower your cores using multiple connections to postgresql)
    7. inherit parent table
    8. now ready to select! In such way I have achieve 400000-500000 inserts per seconds with index creation on a 10 columns (2 xeon, 24 cores, 24 Gb of memory, SSD).

    Bonus: in separete thread remove old data (some_data_X with min X): huge circular buffer with indexing!

提交回复
热议问题