it seems that if you change the to_contacts to something like below, your problem will be solved:
to_contacts = db.relationship('Contact',
secondary='ContactRelation',
primaryjoin='id==contactrelation.from_contact_id',
secondaryjoin='id==contactrelation.to_contact_id',
backref='from_contacts')