The task of separation of the speakers is not a speech recognition task, it's a speaker recognition task. In the speech comminity this task is also known as speaker diarization. There are several packages for speaker diarization and speaker recognition available for Python:
SIDEKIT from LIUM
Bob toolkit from Idiap
Speaker diarization from ISCI
In case you are not restricted to Python, there are others:
LIUM speaker diarization
Speaker recognition setup in Kaldi. Includes state of the art DNN-based i-vectors called x-vectors.