I am having trouble keeping game objects inside of a contained space. When they reach the edge, there is some momentary push back but then they will go right through the wal
I have a pinball prototype that also gave me much trouble in the same areas. These are all the steps I've taken to almost (but not yet entirely) solve these problems:
For fast moving objects:
Set the rigidbody's Interpolate to 'Interpolate' (this does not affect the actual physics simulation, but updates the rendering of the object properly - use this only on important objects from a rendering point of view, like the player, or a pinball, but not for projectiles)
Set Collision Detection to Continuous Dynamic
Attach the script DontGoThroughThings (https://www.auto.tuwien.ac.at/wordpress/?p=260) to your object. This script cleverly uses the Raycasting solution I posted in my other answer to pull back offending objects to before the collision points.
In Edit -> Project Settings -> Physics:
Set Min Penetration for Penalty to a very low value. I've set mine to 0.001
Set Solver Iteration Count to a higher value. I've set mine to 50, but you can probably do ok with much less.
All that is going to have a penalty in performace, but that's unavoidable. The defaults values are soft on performance but are not really intented for proper simulation of small and fast-moving objects.