Spark ignoreCorruptFiles config is ignoring non-corrupt files when first file is corrupt

Question

I have a folder which contains one corrupt file and other proper files(Non-corrupt)

Path: s3://temp-bucket/test/
Files:

s3://temp-bucket/test/a_corrupt_fil