adb devices -l command prints out some additional device parameters which can also be used with adb -s. Obviously it would help only if any of those extra parameters are unique. Fortunately for linux and osx users adb devices -l in those systems also reports unique USB port numbers:
$ adb devices -l
List of devices attached
XXXXXXXXXX device usb:2-1.6.7.7 product:shamu model:Nexus_6 device:shamu
$ adb -s usb:2-1.6.7.7 shell
shell@shamu:/ $
