Importing JSON data and flattening records / populating tables with PostgreSQL

问题

To begin, I'll refer to importing a JSON file in PostgreSQL. It's a neat way of importing JSON data and setting it in a database for further work. Here's a preview:

^{The simplest way to import json from a file appear to not import a single json from a file but rather a single column csv: A list of one-line jsons:}
{"id": 23635,"name": "Jerry Green","comment": "Imported from facebook."}
{"id": 23636,"name": "John Wayne","comment": "Imported from facebook."}
^{then, under psql:}
create table t ( j jsonb )
\copy t from 'd:\path\data.json'
^{One record per json (line) will be added into t table.}

NOTE: COPY will not recognize escaped double quotes \" so importing data with strings containing double quotes will fail without proper parameters. CSV mode will do the job.

copy t (j jsonb) 
from '/path/to/file.json' 
csv quote e'\x01' delimiter e'\x02';

^{more @adpgtech & I II}

What I would like to achieve is to extract data from JSON and flatten it into tables. Like this:

   id  |     name    |         comment
-------+-------------+-------------------------
 23635 | Jerry Green | Imported from facebook.
 23636 | John Wayne  | Imported from facebook.

I commence with queries similar to:

SELECT j->>'id' AS "id"
      ,j->>'name' AS "name"
      ,j->>'comment' AS "comment"
FROM t;

That worked - see my case based on same example on rextester showing what I've been up to.

And here are my questions:
(1) How to build the same table with less and simplified code making it less prone to errors when working with complex JSON structure and large sets of data?

Handling JSON objects with jsonb_to_record(jsonb) is probably a solution but I haven't been able to apply it on records with multiple levels. I found that^{how-to-get-a-json-object-as-column-in-postgresql} and that^{how-to-convert-postgresql-9-4-jsonb-to-object}, both solid directions to start off with.

Update

Extraction with jsonb_to_record():

SELECT d.*
FROM t
    ,jsonb_to_record(t.j) d(id int, name text, comment text, soc_prof_data jsonb);

Full extraction with jsonb_populate_record():

CREATE TEMP TABLE objo(id int, name text, comment text);
CREATE TEMP TABLE obji(tw int, fb int, inst int, lnin int, yt int);
CREATE TEMP TABLE objii(note text);
CREATE TEMP TABLE objiii(depth int,status text,since date);
SELECT d0.*, d1.*, d2.*, d3.*
FROM t
   ,jsonb_populate_record(null::objo, t.j) d0
   ,jsonb_populate_record(null::obji, t.j->'soc_prof_data') d1
   ,jsonb_populate_record(null::objii, t.j->'soc_prof_data'->'inf') d2
   ,jsonb_populate_record(null::objiii, t.j->'soc_prof_data'->'inf'->'lvl') d3;

Populating a table:

CREATE TABLE persinfo(id_pers int PRIMARY KEY,name text,comment text);
INSERT INTO persinfo
SELECT d0.*
FROM t
   ,jsonb_populate_record(null::objo, t.j) d0;

(2) How to manage JSON arrays if they are present in records? How should jsonb_to_recordset(jsonb) be integrated in queries to process data correctly?

Another approach for flattening JSON elements into a table which is just crazy and answers greatly my question (it doesn't address the situation with array elements though).
However, I would like to construct and tweak my own queries, adaptable for different cases. And my basic sql background would probably not be enough to write or modify a function like that.

(3) What would be the best way to create and associate two tables with a foreign key (for those records in soc_prof_data)? I'm not aiming for on-the-fly solution. I believe creating views is a necessary step.

Update

CREATE TABLE socialprofile(id_prof int PRIMARY KEY REFERENCES persinfo,tw int, fb int, inst int, lnin int,yt int);
INSERT INTO socialprofile(id_prof,tw,fb,inst,lnin,yt)
SELECT id_pers, d1.*
FROM persinfo p
   ,t
   ,jsonb_populate_record(null::obji, t.jdata->'soc_prof_data') d1
WHERE p.id_pers = (t.jdata->>'id')::int;

See the forked solution Importing JSON records to populate tables in a PostgreSQL database of the above example.

Is there a way to do this smoother or better?

Any bit of advice for any of the questions would greatly accelerate my day. Thanks !

_{I don't have enough reputation to ask questions in comments. Neither I couldn't insert a preview image of another topic for simplified description of certain details. Please accept my excuses in advance if this should be tagged as duplicate.}

来源：https://stackoverflow.com/questions/58992119/importing-json-data-and-flattening-records-populating-tables-with-postgresql

标签

json

postgresql

database-design

import

flatten