I have created a website and placed it on github in the repo with USRNAME.github.com. First when I went to that website, my website displayed correctly. Then I bought a domain
You need to setup an A record which points to the github server. You said you used gandi.net, so basically you would change your DNS settings for the top level domain by adding(or modifying) an A record to point to github.
If you want, you can add a subdomain (like www.) by adding a CNAME in your DNS to point to the top level domain or an A record which, again, points to github.