It is easy to go to the last edit location in the current buffer. See How to go back to lines edited before the last one in Vim? The changelist is buffer local, each buffer
You could do :windo normal
`.
That said, I usually just use C-o (repeatedly).
If I "feel" that I will likely want to return to some point, I'll just hit mA
(which records a cross-file/buffer mark) so I can just do `A from anywhere (even after restarting the editor).
Slightly off-topic, I love :Obsession
(by Tim Pope) for really long-lived sessions that do a lot of cross-reference navigation.