ignore axis labels when computing margins in patchwork and ggplot2

Question

I\'m trying to create a scatterplot matrix using ggplot2 and patchwork.

My function for this is

library(tidyverse)
library(patchwork)
library(grid)

gg