I was wondering if system endianness matters when converting a byte array to a short / int / long. Would this be incorrect to do if the code runs on both big-endian and litt
The problem as you've specified, where you are using an existing byte array, will work fine across all machines. You will end up with the same answer.
However, depending on how you are creating that stream, it may be affected by endianness and you may not end up with the number you think you will.