Being able to speak from actual experience (AROS, the biggest obstacles are:
- Hen and egg (no os <-> no apps <-> no users <-> can't attract developers)
- Why even try to compete against Windows? Apple doesn't succeed and they have lots of money and Linux doesn't succeed despite having an enormous and enthusiastic community.
- Big companies aren't interested at all to bring their apps to your OS (Adobe Acrobat Reader, MS Office, Macromedia Flash, Java from Sun). Without these, and no good OSS alternatives, you can't attract users.
- It takes very long. In my case, it's been roughly 15 years to get a 1.0 (and we're not 100% there, yet).
- Compatibility. In order to get any users, you must be compatible to something that exists (so people can continue to use their data, etc). If you're compatible, why change to your OS?
So if you plan to write your own OS, you should consider this:
- It will take a long time
- For the longest time, you will be alone. I was lucky because I had this enormously fanatic and dedicated Amiga community which just wouldn't let up.
- You must find a niche where you can offer a service which no other OS can offer.
- People who can do this often get better job offers because the perception is "writing OSs is hard" :)
- You will be a member of a very small, elite group of programmers that can say "I wrote my own OS and it can do more than print 'Hello world!'"