Have you tried simple, off-the-shelf solutions like ImagXpress from Accusoft Pegasus?
The BlobRemoval method can be tuned for pixel count and density to find hole punches, even if they are not continuous. (you can also try a Dilate function to close gaps)
With a little playing around you likely can get the results you need in the vast majority of cases, with very little code or science.
C#:
public void DocumentBlobRemoval(
Rectangle Area,
int MinPixelCount,
int MaxPixelCount,
short MinDensity
)
