I am working on an image gallery on a legacy site. I can\'t use anything other than regular old HTML/jQuery or JS/CSS. I know this would be a lot easier with Bootstrap or so
For this layout use CSS grid. Your original problem won't even exist then. You won't need any workaround or hacks, JavaScript, or any weird stuff to make it work. With CSS grid layout you'll easily achieve what you need with elegant and maintainable code.
CSS grid layout is a kind of new CSS feature, and it's actually the first legit way of doing layout, which is awesome. Read about it:
For your case, especially take a look at the auto-fill or auto-fit properties. Here's another useful link: https://gridbyexample.com/examples/example37/.
And here's a quick example of what you want to achieve using CSS grid: https://codepen.io/anon/pen/YxbexQ?editors=1100. See how it react to different screen/container width. It just works.