In my case in addition to using AVAudioSessionCategoryOptionDefaultToSpeaker option I had to remove changing audioSession mode from AVAudioSessionModeDefault to AVAudioSessionModeMeasurement.
So if you are using modes other than AVAudioSessionModeDefault, be careful.