POST Data with .net Html Agility Pack

Daniel Banck — Sat, 09 Jan 2010 00:19:53 +0000

I’m using the Html Agility Pack for getting and parsing HTML pages. It offers many possibilities, including XPath selectors.

Today I had a problem with posting data to a webpage. It’s possible to call the HtmlWeb Load function with a second “POST” parameter, but it offers no easy way to add values to be posted.
After a while of searching I came across this solution: add this two functions to the HtmlWeb class. (You can get the source in download section, too)

public HtmlDocument SubmitFormValues (NameValueCollection fv, string url)
{
	// Attach a temporary delegate to handle attaching
	// the post back data
	PreRequestHandler handler = delegate(HttpWebRequest request) {
		string payload = this.AssemblePostPayload (fv);
			byte[] buff = Encoding.ASCII.GetBytes (payload.ToCharArray ());
			request.ContentLength = buff.Length;
			request.ContentType = "application/x-www-form-urlencoded";
			System.IO.Stream reqStream = request.GetRequestStream ();
			reqStream.Write (buff, 0, buff.Length);
			return true;
	}
	this.PreRequest += handler;
	HtmlDocument doc = this.Load (url, "POST");
	this.PreRequest -= handler;
	return doc;
}
 
private string AssemblePostPayload (NameValueCollection fv)
{
	StringBuilder sb = new StringBuilder ();
	foreach (String key in fv.AllKeys) {
		sb.Append ("&" + key + "=" + fv.Get (key));
	}
	return sb.ToString ().Substring (1);
}

Source: http://htmlagilitypack.codeplex.com/Thread/View.aspx?ThreadId=14255

Now posting is easy. Example:

NameValueCollection postData = new NameValueCollection (1);
postData.Add ("name", "value");
doc = this.hw.SubmitFormValues (postData, url);

(hw is an HtmlWeb object and doc an HtmlDocument object)

If you need to use cookies, too, you need to modify the HtmlWeb a little more, because there is a little bug.
First add a new private variable:

private CookieContainer _ck = new CookieContainer();

Then find this section:

if (_useCookies) 
{
        req.CookieContainer = new CookieContainer();
}

And change it into:

if (_useCookies) 
{
        req.CookieContainer = this._ck;
}

Now you just have to activate the usage of cookies in your HtmlWeb object:

this.hw = new HtmlWeb ();
this.hw.UseCookies = true;

That’s it! Enjoy easy form posting with a full cookie jar (:

Daniel Banck » post data

POST Data with .net Html Agility Pack