I have a sphere represented in object space by a center point and a radius. The sphere is transformed into world space with a transformation matrix that may include scales, rota
This will not work for non-uniform scaling. It is possible to calculate for arbitrary invertible affine transform with Lagrange multipliers (KKT theorem) and I believe it will get ugly.
However - are you sure you need an exact AABB? You can approximate it by transforming the original AABB of the sphere and getting its AABB. It is larger than the exact AABB so it might fit your application.
For this we need to have three pseudo-functions:
GetAABB(sphere) will get the AABB of a sphere.
GetAABB(points-list) will get the AABB of the given set of points (just the min/max coordinates over all points).
GetAABBCorners(p, q) will get all the 8 corner points of an AABB (p and q are among them).
(p, q) = GetAABB(sphere);
V = GetAABBCorners(p, q);
for i = 1 to 8 do
V[i] = Transform(T, V[i]);
(p, q) = GetAABB(V);