I have an old Windows.Forms Application that I am trying to debug.
Sometimes after running a few minutes it will produce an ArithmeticException or an OverflowExcepti
The diagnostic here is that you have legacy unmanaged code in your process, judging from the call stack you posted that's likely to be an old ActiveX control.
These exceptions are hardware exceptions generated by the FPU, the floating point processor. Which can be put in an operation mode where it reports problems by raising exceptions, like the STATUS_FLOAT_OVERFLOW and STATUS_FLOAT_INVALID_OPERATION exceptions that you are seeing. Instead of generating infinity, NaN or denormals. The FMUL instruction can easily generate such an exception.
Software that changes the FPU operation mode is pretty fundamentally incompatible with managed code. Which requires that FPU exceptions are always masked. Masking these exceptions is entirely normal and what is done with all modern software. Back in the previous century these exceptions were however considered an asset to diagnose floating point calculations going haywire. In particular, old Borland runtime libraries unmasked these exceptions.
Well, this is all rather bad news in case you didn't get that message yet. First place to look is to try to diagnose why this code is throwing floating point exceptions. Bad data tends to be the most common reason. Secondly, you really have to do something about the FPU control register being changed, this can easily cause managed code to fail as well. Particularly a problem in WPF code, it likes using NaN.
Finding such code is pretty easy with the debugger.  Use the Debug + Windows + Registers debugger window.  Right-click the window and tick the "Floating point" option.  The value of the CTRL register is crucial, it should be 027F in a managed program.  Step through the program, coarse at first, you found the trouble-maker when the register changes.  If it is 64-bit program then also tick "SSE", the MXCSR register should be 00001F80.
You cannot directly reset the FPU control register with managed code but you can use a trick. The CLR resets it whenever it handles an exception. So a possible fix is to intentionally throw and catch an exception, after the statement that caused the control register value to change:
        try {  throw new Exception("Resetting FPU control register, please ignore"); }
        catch { }
Pinvoking the _controlfp() function in msvcrt.dll is a more direct way. But of course with the side-effect of both that now that library is operating in a mode that it wasn't designed for, it of course won't expect to encounter Nan and Infinity values. Long term, you really need to consider retiring that old component or library.