I am trying to understand how I can make changes to the internals of a ggplot2 chart. I started reading the few ressources I could find about ggplot_built and <
Build the plot to have all the waiver() stuff (axis limits and labels, etc.) resolve, tamper with that, then convert it to a gtable for plotting.
p_built <- ggplot_build(p)
p_built$plot$layers[[1]] <- NULL
p_built$data[[1]] <- NULL
plot(ggplot_gtable(p_built))
Note that you need to remove not only the layer but also the dataset for that layer.