Bug in PostgreSQL locking mechanism or misunderstanding of the mechanism

Question

We encountered an issue with PostgreSQL 9.0.12 locking mechanism.

This is our minimal code to reproduce the issue:

Scenario

Transaction 1      Transaction 2
BEGIN              BEGIN
......             select trees for update;                
update apples;      
--passes
update apples;    
-- stuck!      

reproduce code: If you want to try it in your PostgreSQL - here is a code you can copy/paste.

I have a following db schema:

CREATE TABLE trees (
    id       integer primary key
);

create table apples (
    id       integer primary key,
    tree_id  integer references trees(id)
);

insert into trees values(1);
insert into apples values(1,1);

open two psql shells:

on shell 1:

BEGIN;
    SELECT id FROM trees WHERE id = 1 FOR UPDATE;    

on shell 2:

BEGIN;
UPDATE apples SET id = id WHERE id = 1;
UPDATE apples SET id = id WHERE id = 1;

The second update of apples will stuck and it seems that the porcess of shell 2 is wating on the transaction of shell 1 to finish.

relname  |transactionid|procpid|mode              |substr                                    |       age      |procpid
-----------+-------------+-------+------------------+------------------------------------------+----------------+-------
           |             | 4911  | ExclusiveLock    | <IDLE> in transaction                    | 00:05:42.718051|4911
           |   190839904 | 4911  | ExclusiveLock    | <IDLE> in transaction                    | 00:05:42.718051|4911
trees      |             | 4911  | RowShareLock     | <IDLE> in transaction                    | 00:05:42.718051|4911
           |             | 5111  | ExclusiveLock    | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
           |   190839905 | 5111  | ExclusiveLock    | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
apples_pkey|             | 5111  | RowExclusiveLock | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
apples     |             | 5111  | RowExclusiveLock | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
trees      |             | 5111  | RowShareLock     | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
trees      |             | 5111  | ShareLock        | UPDATE apples SET id = id WHERE id = 1;  | 00:05:21.67203 |5111
           |             | 2369  | ExclusiveLock    | <IDLE> in transaction                    | 00:00:00.199268|2369
           |             | 2369  | ExclusiveLock    | <IDLE> in transaction                    | 00:00:00.199268|2369
           |             | 5226  | ExclusiveLock    | select pg_class.relname,pg_locks.transac | 00:00:00       |5226

Have we misunderstood something or it is a bug in postgres?

Answer 1

There is no bug, and I don't think you're misunderstanding anything; you're just missing a couple of pieces of the puzzle.

Foreign keys are implemented internally using row-level locking; starting from Postgres 8.1 and up to 9.2, whenever you update the referencing table (apples in this case), a query is fired that does SELECT FOR SHARE on the referenced table (trees). So that SELECT FOR UPDATE in the first transaction blocks the SELECT FOR SHARE of the referential integrity for the second transaction. This is what causes the block in the second command.

Now I hear you yell, “Wait! How come it blocks on the second command and not the first? The explanation is simple, really -- that's just because there is a simple optimization that skips the internal SELECT FOR SHARE when the key is not being modified. However, this is simplistic in that if you update a tuple a second time, this optimization will not fire because it's harder to track down the original values. Hence the blockage.

You might also be wondering why I said this is up to 9.2 --- what's with 9.3? The main difference there is that in 9.3 it uses SELECT FOR KEY SHARE, which is a new, lighter lock level; it allows for better concurrency. If you try your example in 9.3 and also change the SELECT FOR UPDATE to SELECT FOR NO KEY UPDATE (which is a lighter mode than SELECT FOR UPDATE that says you are maybe going to update the tuple, but you promise to not modify the primary key and promise not to delete it), you should see it doesn't block. (Also, you can try an UPDATE on the referenced row and if you don't modify the primary key, then it will also not block.)

This 9.3 stuff was introduced by a patch by yours truly as http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=0ac5ad5134f2769ccbaefec73844f8504c4d6182 and I think it was a pretty cool hack (The commit message has some more details, if you care about that sort of stuff). But beware, do not use versions prior to 9.3.4 because that patch was so hugely complex that a few serious bugs went unnoticed and we only fixed recently.