I want to add/remove servers in my nginx running inside a docker container
I use ADD command in Dockerfile to add my nginx.conf to /etc/nginx dir.
#
If you want to do this from within a container, you will need service/configuration discovery. Tools like coreOS/etcd and Apache Zookeeper are made to facilitate this process. Fair warning: for simple applications, these can be a bit difficult to manage.
There is also docker-gen, which is somewhat easier to get going. There is even a specific pre-made script for exactly the scenario you are describing.