Insufficient Resources error while inserting into SQL table using Vertica

问题

I'm running a Python script to load data from a DataFrame into a SQL Table. However, the insert command is throwing this error:

(pyodbc.Error) ('HY000', '[HY000] ERROR 3587:  Insufficient resources to execute plan on pool fastlane [Request exceeds session memory cap: 28357027KB > 20971520KB]\n (3587) (SQLExecDirectW)')

This is my code:

df.to_sql('TableName',engine,schema='trw',if_exists='append',index=False) #copying data from Dataframe df to a SQL Table

回答1:

Can you do the following for me:

run this command - and share the output. MAXMEMORYSIZE, MEMORYSIZE and MAXQUERYMEMORYSIZE, plus PLANNEDCONCURRENCY give you an idea of the (memory) budget at the time when the query / copy command was planned.

gessnerm@gessnerm-HP-ZBook-15-G3:~/1/fam/fam-ostschweiz$ vsql -x -c \
 "select * from resource_pools where name='fastlane'"
-[ RECORD 1 ]------------+------------------
pool_id                  | 45035996273841188
name                     | fastlane
is_internal              | f
memorysize               | 0%
maxmemorysize            | 
maxquerymemorysize       | 
executionparallelism     | 16
priority                 | 0
runtimepriority          | MEDIUM
runtimeprioritythreshold | 2
queuetimeout             | 00:05
plannedconcurrency       | 2
maxconcurrency           | 
runtimecap               | 
singleinitiator          | f
cpuaffinityset           | 
cpuaffinitymode          | ANY
cascadeto                |

Then, you should dig, out of the QUERY_REQUESTS system table, the acutal SQL command that your python script triggered. It should be in the format of:

COPY <_the_target_table_> 
FROM STDIN DELIMITER ',' ENCLOSED BY '"' 
DIRECT REJECTED DATA '<_bad_file_name_>'

or similar.

Then: how big is the file / are the files you're trying to load in one go? If too big, then B.Muthamizhselvi is right - you'll need to portion the data volume you load.

Can you also run:

vsql -c "SELECT EXPORT_OBJECTS('','<schema>.<table>',FALSE)"

.. .and share the output? It could well be that you have too many projections for the memory to be enough, that you are sorting by too many columns.

Hope this helps for starters ...

来源：https://stackoverflow.com/questions/57478624/insufficient-resources-error-while-inserting-into-sql-table-using-vertica

标签

sql

vertica