Changing database per view & accessing multiple databases per view

Question

I'm having some problems using SQLAlchemy in Pyramid. Although I can find examples of what I need, they're normally very short and lacking. So I've ended up with patchy code that barely makes any sense. So I'm hoping someone could give a fuller example of what I need to do.

I have 4 databases all with the same schema. I want to be able to work on them from one Pyramid app, sometimes listing all "orders" from all 4 databases, sometimes just listing all "orders" from "site1". As the schemas are the same, I also use the same model classes for the databases.

I've tried it with both sqlahelper and plain SQLAlchemy with no luck. The code below uses sqlahelper but I'm happy to use anything that works:

__init__.py

site1_eng = engine_from_config(settings, prefix='site1.')
site2_eng = engine_from_config(settings, prefix='site2.')
site3_eng = engine_from_config(settings, prefix='site3.')

sqlahelper.add_engine(site1_eng, 'site1_eng')
sqlahelper.add_engine(site2_eng, 'site2_eng')

views.py

def site_orders(request):
    site = request.matchdict['site']
    db_eng = sqlahelper.get_engine(("%s_eng" % (site)))
    conn = db_eng.connect()
    dbsession = sqlahelper.get_session()
    dbsession.configure(bind=conn)

    orders = dbsession.query(Order).order_by(Order.cdate.desc())[:100]

    return dict(orders=orders, pagetitle=(site+" Orders"))

What Happens?

Well I'd hoped it would switch database depending on the URL and it does! However, it seems completely random as to which is chooses. So /orders/site1/ will sometimes go to site2 database and sometimes site3. Refreshing will often switch the database it chooses each time. Same for other URL's.

Its almost as if the session isn't binding to the database and its picking whichever happens to be in the session at the time? That may not make sense - my understanding of SQLAlchemy isn't great.

Really hope someone can help as it all hinges on the ability to quickly and easily switch databases within a view and at the moment it seems completely impossible to control it.

NOTE: I did originally try following and altering the Pyramid SQLA+URL Dispatcher tutorial which used:

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))

But I removed that when finding sqlahelper. If I should be using it let me know.

Answer 1

Configuring and connection for each request seems like a lot of work to me. I would create four session handlers in my model module and just choose from them.

Example:

models/__init__.py

DBSession1 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))  
DBSession2 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))  
DBSession3 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))  
DBSession4 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))  
metadata1 = MetaData()                                                           
metadata2 = MetaData()                                                           
metadata3 = MetaData()                                                           
metadata4 = MetaData()                                                           

def initialize_sql(engines, drop_db=False):                                      
    DBSession1.configure(bind=engine[0])
    DBSession2.configure(bind=engine[1])                                            
    DBSession3.configure(bind=engine[2])                                            
    DBSession4.configure(bind=engine[3])                                            
    metadata1.bind = engine[0]                                            
    metadata2.bind = engine[1]                                            
    metadata3.bind = engine[2]                                            
    metadata4.bind = engine[3]  

and then in your view:

from mypackage.models import DBSession1, DBSession2, DBSession3, DBSession4

def site_orders(request)                                                      
    site = request.matchdict['site']                                                      
    dbsession = globals().get("DBSession%d" % site)                                                      
    orders = dbsession.query(Order).order_by(Order.cdate.desc())[:100]                                                      
    return dict(orders=orders, pagetitle=(site+" Orders"))

Answer 2

You can set engine to the sqlalchemy session directly

Example for listing all "orders" from all 4 databases:

def site_orders(request):
    ...
    orders = []
    for engine in engines:
        dbsession.bind = engine
        orders += dbsession.query(Order).order_by(Order.cdate.desc())[:100]

    return dict(orders=orders, pagetitle=(site+" Orders"))