Refactor SQLite Table by splitting it in two and link with foreign keys

Question

I'm working on a SQLite Database. The database is already filled, but I want to refactor it. Here is a sample of what I need to do:

I currently have one table:

CREATE TABLE Cars (ID INTEGER PRIMARY KEY,
                   Name VARCHAR(32),
                   TopSpeed FLOAT,                   
                   EngineCap FLOAT);

I want to split this into two tables:

CREATE TABLE Vehicles (ID INTEGER PRIMARY KEY,
                       Name VARCHAR(32),
                       TopSpeed FLOAT); 

CREATE TABLE Cars (ID INTEGER PRIMARY KEY,
                   VehicleID INTEGER CONSTRAINT FK_Cars REFERENCES [Vehicles](ID),                  
                   EngineCap FLOAT);          

I have figured out to create a temporary table with the Cars table contents, and I can fill up the Vehicles table with the contents of the Cars table:

CREATE TEMPORARY TABLE Cars_temp AS SELECT * FROM Cars;

INSERT INTO Vehicles (Name, TopSpeed)
SELECT Name, TopSpeed FROM Cars_temp;

But I am still looking for a way to go over that same selection, while putting the EngineCap field into the new Cars table and somehow extracting the corresponding ID value from the Vehicles table to put into the VehicleID foreign key field on the Cars table.

I'm open for workaround or alternative approaches.

Thanks.

Answer 1

Simple solution without triggers:

• create VEHICLES_TEMP table including the CAR_ID
• create your new CARS table without the VEHICLES columns you don't want
• update CARS with VEHICLE_ID taken from VEHICLES_TEMP (identified by the CAR_ID)
• create final VEHICLES table without the CAR_ID

Answer 2

Since @mateusza did not provide an example, I've made one:

Suppose you have this table:

CREATE TABLE [Customer] (
  [name] TEXT,
  [street] TEXT,
  [city] TEXT);

Now you want to move street and city into a separate table Address, so you'll end up with two tables:

CREATE TABLE [Customer2] (
  [name] TEXT,
  [addr] INTEGER);

CREATE TABLE [Address] (
  [rowid] INTEGER NOT NULL,
  [street] TEXT,
  [city] TEXT,
  PRIMARY KEY ([rowid])
);

(For this example, I'm doing the conversion in the same database. You'd probably use two DBs, converting one into the other, with an SQL ATTACH command.)

Now we create a view (which imitates our original table using the new tables) and the trigger:

CREATE VIEW Customer1 (name, street, city) AS
    SELECT C.name, A.street, A.city FROM Customer2 AS C
    JOIN Address as A ON (C.addr == A.rowid);

CREATE TEMP TRIGGER TempTrig INSTEAD OF INSERT ON Customer1 FOR EACH ROW BEGIN
    INSERT INTO Address (street, city) SELECT NEW.street, NEW.city;
    INSERT INTO Customer2 (addr, name) SELECT last_insert_rowid(), NEW.name;
END;

Now you can copy the table rows:

INSERT INTO Customer1 (name, street, city) SELECT name, street, city FROM Customer;

---

The above is a simplified case where you'd only move some data into a single new table.

A more complex (and more general) case is where you want to...

1. Separate your original table's columns into several foreign tables, and
2. Have unique entries in the foreign tables (that's usually the reason why you'd refactor your table).

This adds some additional challenges:

1. You'll end up inserting into multiple tables before you can insert their rowids into the table with the referencing rowids. This requires storing the results of each INSERT's last_insert_rowid() into a temporary table.
2. If the value already exists in the foreign table, its rowid must be stored instead of the one from the (non-executed) insertion operation.

Here's a complete solution for this. It manages a database of music records, constisting of a song's name, album title and artist name.

-- Original table
CREATE TABLE [Song] (
  [title] TEXT,
  [album] TEXT,
  [artist] TEXT
);

-- Refactored tables
CREATE TABLE [Song2] (
  [title] TEXT,
  [album_rowid] INTEGER,
  [artist_rowid] INTEGER
);
CREATE TABLE [Album] (
  [rowid] INTEGER PRIMARY KEY AUTOINCREMENT,
  [title] TEXT UNIQUE
);
CREATE TABLE [Artist] (
  [rowid] INTEGER PRIMARY KEY AUTOINCREMENT,
  [name] TEXT UNIQUE
);

-- Fill with sample data

INSERT INTO Song VALUES ("Hunting Girl", "Songs From The Wood", "Jethro Tull");
INSERT INTO Song VALUES ("Acres Wild", "Heavy Horses", "Jethro Tull");
INSERT INTO Song VALUES ("Broadford Bazar", "Heavy Horses", "Jethro Tull");
INSERT INTO Song VALUES ("Statue of Liberty", "White Music", "XTC");
INSERT INTO Song VALUES ("Standing In For Joe", "Wasp Star", "XTC");
INSERT INTO Song VALUES ("Velvet Green", "Songs From The Wood", "Jethro Tull");

-- Conversion starts here

CREATE TEMP TABLE [TempRowIDs] (
  [album_id] INTEGER,
  [artist_id] INTEGER
);

CREATE VIEW Song1 (title, album, artist) AS
  SELECT Song2.title, Album.title, Artist.name
    FROM Song2
    JOIN Album ON (Song2.album_rowid == Album.rowid)
    JOIN Artist ON (Song2.artist_rowid == Artist.rowid);

CREATE TEMP TRIGGER TempTrig INSTEAD OF INSERT ON Song1 FOR EACH ROW BEGIN
  INSERT OR IGNORE INTO Album (title) SELECT NEW.album;
  UPDATE TempRowIDs SET album_id = (SELECT COALESCE (
    (SELECT rowid FROM Album WHERE changes()==0 AND title==NEW.album), last_insert_rowid()
  ) ) WHERE rowid==1;
  INSERT OR IGNORE INTO Artist (name) SELECT NEW.artist;
  UPDATE TempRowIDs SET artist_id = (SELECT COALESCE (
    (SELECT rowid FROM Artist WHERE changes()==0 AND name==NEW.artist), last_insert_rowid()
  ) ) WHERE rowid==1;
  INSERT INTO Song2 (title, album_rowid, artist_rowid) SELECT
    NEW.title, (SELECT album_id FROM TempRowIDs), (SELECT artist_id FROM TempRowIDs);
END;

INSERT INTO TempRowIDs DEFAULT VALUES;

INSERT INTO Song1 (title, album, artist) SELECT title, album, artist FROM Song;

DROP TRIGGER TempTrig;
DROP TABLE TempRowIDs;

-- Conversion ends here

-- Print results
SELECT * FROM Song;
SELECT * FROM Song1;

-- Check if original and copy are identical (https://stackoverflow.com/a/13865679/43615)
SELECT CASE WHEN (SELECT COUNT(*) FROM (SELECT * FROM Song UNION SELECT * FROM Song1)) == (SELECT COUNT() FROM Song) THEN 'Success' ELSE 'Failure' END;

Note that this example has one potential issue: If the constraints on the foreign table are more complex, the SELECT rowid FROM search for the existing entry needs to be updated accordingly. Ideally, SQLite should provide a way to determine the conflicting rowid somehow, but it doesn't, unfortunately (see this related question).

Answer 3

Create a table New_Cars and a INSTEAD OF INSERT trigger, which will insert data to both tables Vehicles and Cars. When inserting to Cars, you can use last_insert_rowid() function to refer to inserted row in Vehicles table.

This can be temporary solution, or you can leave it in your database for further modifications.