I know a little about assembly, and that there are 4 or 8 or so general purpose registers. How do all the programs on a computer work with just that amount of registers, especia
That's one of the things that the computer's other storage, particularly RAM, is used for: to save and restore bits of data in and out of registers.
When a thread is switched away so another thread can run. the first threads register state is saved somewhere (on the stack or some other data structure), and the seconds thread's register state is restored from wherever it was saved before. RAM is fast enough that thousands of these switches can occur in a second, but takes enough time that if you're swapping threads needlessly it can impact performance.
Another very, very common occurrence is local variables - if a local variable is used for a short enough period of time, it may never exist outside of a register. however, in many cases, a local variable might need to be saved from the register to a memory location so some other value can be loaded into and manipulated in a register. The same actually happens for pretty much any variables, not just locals (but it's much more likely for a local to never have an existence in memory).