Keep .git folder size low while tracking very big files

Question

I am trying to set up a local git for a 150Gb folder. I could finally make it work, but the .git folder in that repository also grew up to a bit over 100Gb, mostly from obje