I am trying to write selenium tests for a website using java. However, I have come across a problem when testing file uploading..
When I click the file upload button
I have answered this for a similar question. There are other solutions provided for Upload - Like using AutoIT. But I personally would defer to interact with any OS specific dialogues. Interacting with OS specific dialogues would limit you to run the tests from a given environment.
Selenium webdriver java - upload file with phantomjs driver
Always identify & interact with elements of type "file" when uploads are concerned. This would solve your issue of pop ups.
Ex: In my application, upload related elements have the below DOM -
In this case, you can use sendKeys method to "multiFileInput" which is of type "file". This way it would work for all FF, Chrome & also headless browsers.