postgresql-copy

Check if records exists in a Postgres table

不羁岁月 提交于 2019-11-30 20:53:38
问题 I have to read a CSV every 20 seconds. Each CSV contains min. of 500 to max. 60000 lines. I have to insert the data in a Postgres table, but before that I need to check if the items have already been inserted, because there is a high probability of getting duplicate item. The field to check for uniqueness is also indexed. So, I read the file in chunks and use the IN clause to get the items already in the database. Is there a better way of doing it? 回答1: This should perform well: CREATE TEMP

Correct way to use copy Postgres jdbc

二次信任 提交于 2019-11-30 19:17:31
问题 Unable to use copy command with jdbc Postgres. Whats wrong with the below code snippet sample. public boolean loadReportToDB(String date) { // TODO Auto-generated method stub Connection connection = DBUtil.getConnection("POSTGRESS"); String fileName = "C:/_0STUFF/NSE_DATA/nseoi_" + date + ".csv"; String sql = "\\copy fno_oi FROM 'C:\\_0STUFF\\NSE_DATA\\nseoi_27102017.csv' DELIMITER ',' CSV header"; try { PreparedStatement ps = connection.prepareStatement(sql); System.out.println("query"+ps

How to convert date strings to timestamp without knowing the date format

限于喜欢 提交于 2019-11-30 12:08:13
I am trying to write a query to insert a value into a timestamp with no timezone data type field. The value is coming from CSV file. The version I am working with is PostgreSQL 8.1.21 . The CSV file upload is done by the client and it has a date column. The date sometimes comes as '28-Sep-13' and sometimes as '28/09/2013' formats. I tried to use the following to cast the string into timestamp: str_date::timestamp . This works fine if str_date is something like '28-Sep-13' but it won't work if the incoming date has the format '28/09/2013' , when this error occurs: ERROR: date/time field value

COPY function in PostgreSQL

感情迁移 提交于 2019-11-30 09:18:10
I would like to use the COPY function in PostgreSQL to import a CSV file into a PostgreSQL database. Where it says the filename in the documentation , does the CSV file have to be stored in a specific location or can it be stored in any location. For example, copy data_table from '/tmp/outputdata.csv' WITH DELIMITER AS ',' CSV QUOTE AS '"'; . Where it says tmp , does that mean the tmp folder in the C: drive. Can it be change to another folder name? It looks like you are confused by Linux vs. Windows file-path notation. What you have there is a Linux path anchored to root. Windows uses drive

COPY function in PostgreSQL

安稳与你 提交于 2019-11-29 14:45:07
问题 I would like to use the COPY function in PostgreSQL to import a CSV file into a PostgreSQL database. Where it says the filename in the documentation, does the CSV file have to be stored in a specific location or can it be stored in any location. For example, copy data_table from '/tmp/outputdata.csv' WITH DELIMITER AS ',' CSV QUOTE AS '"'; . Where it says tmp , does that mean the tmp folder in the C: drive. Can it be change to another folder name? 回答1: It looks like you are confused by Linux

In PostgreSQL, how to insert data with COPY command?

寵の児 提交于 2019-11-29 09:08:59
I have problem when run 1 project NodeJs with PostgreSQL database. I have error when trying to insert data in pgAdmin using the COPY command. COPY beer (name, tags, alcohol, brewery, id, brewery_id, image) FROM stdin; Bons Voeux blonde 9.5 Brasserie Dupont 250 130 generic.png This data in gist : This error: ERROR: syntax error at or near "Bons" SQL state: 42601 Character: 1967 COPY tbl FROM STDIN ; is not supported by pgAdmin. You get a plain syntax error because Postgres gets the data as SQL code. Four possible solutions: 1. Use a multi-row INSERT instead: INSERT INTO beer(name, tags, alcohol

How to ignore errors with psql \\copy meta-command

放肆的年华 提交于 2019-11-29 07:03:39
I am using psql with a PostgreSQL database and the following copy command: \COPY isa (np1, np2, sentence) FROM 'c:\Downloads\isa.txt' WITH DELIMITER '|' I get: ERROR: extra data after last expected column How can I skip the lines with errors? Erwin Brandstetter You cannot skip the errors without skipping the whole command up to and including Postgres 9.5. There is currently no more sophisticated error handling. \copy is just a wrapper around SQL COPY that channels results through psql. The manual for COPY : COPY stops operation at the first error. This should not lead to problems in the event

ERROR: COPY delimiter must be a single one-byte character

爷,独闯天下 提交于 2019-11-29 04:42:19
I want to load the data from a flat file with delimiter "~,~" into a PostgreSQL table. I have tried it as below but looks like there is a restriction for the delimiter. If COPY statement doesn't allow multiple chars for delimiter, is there any alternative to do this? metadb=# \COPY public.CME_DATA_STAGE_TRANS FROM 'E:\Infor\Outbound_Marketing\7.2.1\EM\metadata\pgtrans.log' WITH DELIMITER AS '~,~' ERROR: COPY delimiter must be a single one-byte character \copy: ERROR: COPY delimiter must be a single one-byte character If you are using Vertica , you could use E'\t'or U&'\0009' To indicate a non

Importing zipped CSV file into PostgreSQL

久未见 提交于 2019-11-28 10:22:06
I have a big compressed csv file (25gb) and I want to import it into PostgreSQL 9.5 version. Is there any fast way to import zip or qzip file into postgres without extracting the file? There is an old trick to use a named pipe (works on Unix, don't know about Windows) create a named pipe: mkfifo /tmp/omyfifo write the file contents to it: zcat mycsv.csv.z > /tmp/omyfifo & [from psql] copy mytable(col1,...) from '/tmp/omyfifo' [when finished] : rm /tmp/omyfifo The zcat in the backgound will block until a reader (here: the COPY command) will start reading, and it will finish at EOF. (or if the

How to copy from CSV file to PostgreSQL table with headers in CSV file?

徘徊边缘 提交于 2019-11-28 03:26:24
I want to copy a CSV file to a Postgres table. There are about 100 columns in this table, so I do not want to rewrite them if I don't have to. I am using the \copy table from 'table.csv' delimiter ',' csv; command but without a table created I get ERROR: relation "table" does not exist . If I add a blank table I get no error, but nothing happens. I tried this command two or three times and there was no output or messages, but the table was not updated when I checked it through PGAdmin. Is there a way to import a table with headers included like I am trying to do? G. Cito This worked. The first