I made a simple image through Dockerfile from Fedora (initially 320 MB).
Added Nano (this tiny editor of 1MB size), and the size of the image has risen to 530 MB. I\
Yes, those sizes are ridiculous, and I really have no idea why so few people notice that.
I made an Ubuntu image that is actually minimal (unlike other so-called "minimal" images). It's called textlab/ubuntu-essential and has 60 MB.
FROM textlab/ubuntu-essential
RUN apt-get update && apt-get -y install nano
The above image is 82 MB after installing nano.
FROM textlab/ubuntu-essential
RUN apt-get update && apt-get -y install nano git
Git has many more prerequisites, so the image gets larger, about 192 MB. That's still less that the initial size of most images.
You can also take a look at the script I wrote to make the minimal Ubuntu image for Docker. You can perhaps adapt it to Fedora, but I'm not sure how much you will be able to uninstall.