Changing a matrix from right-handed to left-handed coordinate system

Question

I would like to change a 4x4 matrix from a right handed system where:
x is left and right, y is front and back and z is up and down

to a left-handed system where

Answer 1

It is often the case that you want to change a matrix from one set of forward/right/up conventions to another set of forward/right/up conventions. For example, ROS uses z-up, and Unreal uses y-up. The process works whether or not you need to do a handedness-flip.

Note that the phrase "switch from right-handed to left-handed" is ambiguous. There are many left-handed forward/right/up conventions. For example: forward=z, right=x, up=y; and forward=x, right=y, up=z. You should really think of it as "how do I convert ROS' notion of forward/right/up to Unreal's notion of forward/right/up".

So, it's a straightforward job to create a matrix that converts between conventions. Let's assume we've done that and we now have

mat4x4 unrealFromRos = /* construct this by hand */;
mat4x4 rosFromUnreal = unrealFromRos.inverse();

Let's say the OP has a matrix that comes from ROS, and she wants to use it in Unreal. Her original matrix takes a ROS-style vector, does some stuff to it, and emits a ROS-style vector. She needs a matrix that takes an Unreal-style vector, does the same stuff, and emits an Unreal-style vector. That looks like this:

mat4x4 turnLeft10Degrees_ROS = ...;
mat4x4 turnLeft10Degrees_Unreal = unrealFromRos * turnLeft10Degrees_ROS * rosFromUnreal;

It should be pretty clear why this works. You take a Unreal vector, convert it to ROS-style, and now you can use the ROS-style matrix on it. That gives you a ROS vector, which you convert back to Unreal style.

Gerrit's answer is not quite fully general, because in the general case, rosFromUnreal != unrealFromRos. It's true if you're just inverting a single axis, but not true if you're doing something like converting X->Y, Y->Z, Z->X. I've found that it's less error-prone to always use a matrix and its inverse to do these convention switches, rather than to try to write special functions that flip just the right members.

This kind of matrix operation inverse(M) * X * M comes up a lot. You can think of it as a "change of basis" operation; to learn more about it, see https://en.wikipedia.org/wiki/Matrix_similarity.