How to use httpRequest and HtmlAgilityPack together?

Question

So I am trying to log into a page and get some data out of that page, but my problem is filling in these Login information.

What have I tried? Posting data in url. By simply adding the postname en the value like so: http://stackoverflow.com?postLogin=myUsername&postPassword=myPassword. I sent a request with this url. It gave me an error. I checked out the url on a browser. It seems that it does fill in the username but not the password textbox(I have written the correct postnames).

I also saw someone giving advice to use HtmlAgilityPack to fill in the data(which was marked as answer). But after more searching it seemed impossible to do it with HtmlAgilityPack alone. HtmlAgilityPack is good for parsing html rather than filling it. So I came to a conclusion that I perhaps have to use a combination of httpRequest and HtmlAgilityPack.

I might be wrong, so if you have a better suggestion that would help be free to point me out in the right direction.

But if I am suppose to do it this way, I am not sure how to use both of them together. I first send my webrequest to the login page to get the htmlpage. I get the response from my httpWebRequest, but how do I let my Webbrowser class parse that response and afterwards let my webrequest send in the post data with username, password and of course try to submit it?

If you want to give answer in c# i think i would be able to read it and translate it myself.

edit: I just used fiddler and got 8 posts that are being sent. Does this mean i have sent all 8 aswell to the url ?

Answer 1

There is no need to use WebBrowser for parsing HTML. You can either look for the needed information in the HTML as a string or use library like http://htmlagilitypack.codeplex.com/ to do the parsing.

Also you will find http://www.fiddler2.com/ very useful to determine exactly what information the browser sends to the web page so you can reproduce the behavior using HttpWebRequest (headers, GET/POST values etc.).

You could also try http://www.chadsowald.com/software/fiddler-extension-request-to-code - extension to Fiddler that automatically generates .NET code that imitates the browser request.