We have a series of tables that have grown organically to several million rows, in production doing an insert or update can take up to two seconds. However if I dump the tab
Updating a table requires indices to be rebuilt. If you are doing bulk inserts, try to do them in one transaction (as the dump and restore does). If the table is write-biased I would think about dropping the indices anyway or let a background job do read-processing of the table (eg by copying it to an indexed one).