There is previous little on the google on this subject other than people asking this very same question.
How would I get started writing my own firewall?
This question is alarmingly similar to those asking how to write an encryption algorithm. The answers to both should end in gentle reminders about industry standard solutions that already:
A firewall must inspect every packet efficiently and accurately, and it therefore runs within the OS kernel or network stacks. Errors or inefficiencies jeopardize the security and performance of the entire machine and those downstream.
Building your own low-level firewall is an excellent exercise that will provide an education across many technologies. But for any real application, it's much safer and smarter to build a shell around the existing firewall API. Under Windows, the
netsh command will do this; Linux uses
iptables. Googling any of these will point you to lots of theory, examples, and other helpful information.
So, to get started, I'd brush up on TCP/IP (specifically, the header information: ports and protocols), then learn about the various types of attacks and how to detect them. Learn about each operating system of interest and how it interacts with the network stacks. Finally, think about administration and logging: how will you configure your firewall and trace packets through it to ensure it's doing what you want it to do?
For Windows 2000/XP there is an article with examples on CodeProject Developing Firewalls for Windows 2000/XP
For Vista I think you will need to use Windows Filtering Platform
The usual approach is to use API hooking. Google can teach you that. Just hook all important networking stuff, like
listens's, and refuse what you want.