Connecting to IBM AS400 server for database operations hangs

Question

I\'m trying to talk to an AS400 in Python. The goal is to use SQLAlchemy, but when I couldn\'t get that to work I stepped back to a more basic script using just ibm_db inste

Answer 1

If you don't need Pandas/SQLAlchemy, just use pyodbc as suggested in John Y's answer. Otherwise, you can try doing what worked for me, below. It's taken from my answer to my own, similar question, which you can check out for more detail on what doesn't work (I tried and failed in so many ways before getting it working).

1. I created a blank file in my project to appease this message that I was receiving:

   Unable to open 'hashtable_class_helper.pxi': File not found (file:///c:/git/dashboards/pandas/_libs/hashtable_class_helper.pxi).

   (My project folder is C:/Git/dashboards, so I created the rest of the path.)

2. With that file present, the code below now works for me. For the record, it seems to work regardless of whether the ibm_db_sa module is modified as suggested in John Y's answer, so I would recommend leaving that module alone. Note that although they aren't imported by directly, you need these modules installed: pyodbc, ibm_db_sa, and possibly future (if using Python 2...I forget if it's necessary). If you are using Python 3, I you'll need urllib.parse instead of urllib. I also have i Access 7.1 drivers installed on my computer, which probably came into play.

import urllib
import pandas as pd
from sqlalchemy import create_engine

CONNECTION_STRING = (
    "driver={iSeries Access ODBC Driver};"
    "system=ip_address;"
    "database=database_name;"
    "uid=username;"
    "pwd=password;"
)

SQL= "SELECT..."

quoted = urllib.quote_plus(CONNECTION_STRING)
engine = create_engine('ibm_db_sa+pyodbc:///?odbc_connect={}'.format(quoted))

df = pd.read_sql_query(
    SQL,
    engine,
    index_col='some column'
)
print df

Answer 2

The README for ibm_db_sa only lists DB2 for Linux/Unix/Windows in the "Supported Database" section. So it most likely doesn't work for DB2 for i, at least not right out of the box.

Since you've stated you have IBM System i Access for Windows, I strongly recommend just using one of the drivers that comes with it (ODBC, OLEDB, or ADO.NET, as @Charles mentioned).

Personally, I always use ODBC, with either pyodbc or pypyodbc. Either one works fine. A simple example:

import pyodbc

connection = pyodbc.connect(
    driver='{iSeries Access ODBC Driver}',
    system='11.22.33.44',
    uid='username',
    pwd='password')
c1 = connection.cursor()

c1.execute('select * from qsys2.sysschemas')
for row in c1:
    print row

Now, one of SQLAlchemy's connection methods is pyodbc, so I would think that if you can establish a connection using pyodbc directly, you can somehow configure SQLAlchemy to do the same. But I'm not an SQLAlchemy user myself, so I don't have example code for that.

UPDATE

I managed to get SQLAlchemy to connect to our IBM i and execute straight SQL queries. In other words, to get it to about the same functionality as simply using PyODBC directly. I haven't tested any other SQLAlchemy features. What I did to set up the connection on my Windows 7 machine:

• Install ibm_db_sa as an SQLAlchemy dialect
  You may be able to use pip for this, but I did it the low-tech way:

  1. Download ibm_db_sa from PyPI.
     As of this writing, the latest version is 0.3.2, uploaded on 2014-10-20. It's conceivable that later versions will either be fixed or broken in different ways (so in the future, the modifications I'm about to describe might be unnecessary, or they might not work).
  2. Unpack the archive (ibm_db_sa-0.3.2.tar.gz) and copy the enclosed ibm_db_sa directory into the sqlalchemy\dialects directory.

• Modify sqlalchemy\dialects\ibm_db_sa\pyodbc.py
  

  • Add the initialize() method to the AS400Dialect_pyodbc class
    The point of this is to override the method of the same name in DB2Dialect, which AS400Dialect_pyodbc inherits from. The problem is that DB2Dialect.initialize() tries to set attributes dbms_ver and dbms_name, neither of which is available or relevant when connecting to IBM i using PyODBC (as far as I can tell).
  • Add the module-level name dialect and set it to the AS400Dialect_pyodbc class

Code for the above modifications should go at the end of the file, and look like this:

    def initialize(self, connection):
        super(DB2Dialect, self).initialize(connection)

dialect = AS400Dialect_pyodbc

Note the indentation! Remember, the initialize() method needs to belong to the AS400Dialect_pyodbc class, and dialect needs to be global to the module.

Finally, you need to give the engine creator the right URL:

'ibm_db_sa+pyodbc://username:password@host/*local'

(Obviously, substitute valid values for username, password, and host.)

That's it. At this point, you should be able to create the engine, connect to the i, and execute plain SQL through SQLAlchemy. I would think a lot of the ORM stuff should also work at this point, but I have not verified this.

Answer 3

Here is an example to work with as400, sqlalchemy and pandas. This exammple take a bunch of csv files and insert with pandas/sqlalchemy. Only works for windows, on linux the i series odbc driver segfaults (Centos 7 and Debian 9 x68_64)

Client is Windows 10.

My as400 version is 7.3

Python is 2.7.14

installed with pip: pandas, pyodbc, imb_db_sa, sqlalchemy

You need to install i access for windows from ftp://public.dhe.ibm.com/as400/products/clientaccess/win32/v7r1m0/servicepack/si66062/

Aditionally the modifications by @JohnY on pyodbc.py C:\Python27\Lib\site-packages\sqlalchemy\dialects\ibm_db_sa\pyodbc.py Change line 99 to

    pyodbc_driver_name = "IBM i Access ODBC Driver"

The odbc driver changed it's name.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pandas as pd
import numpy as np
from sqlalchemy import create_engine
import glob

csvfiles=(glob.glob("c:/Users/nahum/Documents/OUT/*.csv"))
df_csvfiles = pd.DataFrame(csvfiles)
for index, row in df_csvfiles.iterrows():
    datastore2=pd.read_csv(str(row[0]), delimiter=',', header=[0],skipfooter=3)
    engine = create_engine('ibm_db_sa+pyodbc://DB2_USER:PASSWORD@IP_SERVER/*local')
    datastore2.to_sql('table', engine, schema='SCHEMA', chunksize=1000, if_exists='append', index=False)

Hope it helps.

Answer 4

The way to find out what port is needed is to look at the service table entries on the IBM i.

Your IBM i guy can use the iNav GUI or the green screen Work with Service Table Entry (WRKSRVTBLE) command

Should get a screen like so:

Service                                  Port  Protocol  

as-admin-http                            2001  tcp       
as-admin-http                            2001  udp       
as-admin-https                           2010  tcp       
as-admin-https                           2010  udp       
as-central                               8470  tcp       
as-central-s                             9470  tcp       
as-database                              8471  tcp       
as-database-s                            9471  tcp  
drda                                      446  tcp
drda                                      446  udp     

The default port for the DB is indeed 8471. Though drda is used for "distributed db" operations.

Based upon this thread, to use ibm_db to connect to DB2 on an IBM i, you need the IBM Connect product; which is a commercial package that has to be paid for.

This thread suggests using ODBC via the pyodbc module. It also suggests that JDBC via the JT400 toolkit may also work.