POST Data with .net Html Agility Pack

Daniel Banck — Sat, 09 Jan 2010 00:19:53 +0000

I’m using the Html Agility Pack for getting and parsing HTML pages. It offers many possibilities, including XPath selectors.

Today I had a problem with posting data to a webpage. It’s possible to call the HtmlWeb Load function with a second “POST” parameter, but it offers no easy way to add values to be posted.
After a while of searching I came across this solution: add this two functions to the HtmlWeb class. (You can get the source in download section, too)

public HtmlDocument SubmitFormValues (NameValueCollection fv, string url)
{
	// Attach a temporary delegate to handle attaching
	// the post back data
	PreRequestHandler handler = delegate(HttpWebRequest request) {
		string payload = this.AssemblePostPayload (fv);
			byte[] buff = Encoding.ASCII.GetBytes (payload.ToCharArray ());
			request.ContentLength = buff.Length;
			request.ContentType = "application/x-www-form-urlencoded";
			System.IO.Stream reqStream = request.GetRequestStream ();
			reqStream.Write (buff, 0, buff.Length);
			return true;
	}
	this.PreRequest += handler;
	HtmlDocument doc = this.Load (url, "POST");
	this.PreRequest -= handler;
	return doc;
}
 
private string AssemblePostPayload (NameValueCollection fv)
{
	StringBuilder sb = new StringBuilder ();
	foreach (String key in fv.AllKeys) {
		sb.Append ("&" + key + "=" + fv.Get (key));
	}
	return sb.ToString ().Substring (1);
}

Source: http://htmlagilitypack.codeplex.com/Thread/View.aspx?ThreadId=14255

Now posting is easy. Example:

NameValueCollection postData = new NameValueCollection (1);
postData.Add ("name", "value");
doc = this.hw.SubmitFormValues (postData, url);

(hw is an HtmlWeb object and doc an HtmlDocument object)

If you need to use cookies, too, you need to modify the HtmlWeb a little more, because there is a little bug.
First add a new private variable:

private CookieContainer _ck = new CookieContainer();

Then find this section:

if (_useCookies) 
{
        req.CookieContainer = new CookieContainer();
}

And change it into:

if (_useCookies) 
{
        req.CookieContainer = this._ck;
}

Now you just have to activate the usage of cookies in your HtmlWeb object:

this.hw = new HtmlWeb ();
this.hw.UseCookies = true;

That’s it! Enjoy easy form posting with a full cookie jar (:

Bug in cherokee

Daniel Banck — Wed, 01 Jul 2009 19:55:41 +0000

If you’re trying to use cherokees ‘Advanced Virtual Hosting’ and a ‘File exists’ rule it will not work.
The rule is not matching any files. Without the AVHosting the rule is working well.

I already reported this bug and it’s fixed in the latest svn version. So if you need to work with AVHosting and this rule you better checkout from svn and compile on your own instead of using any precompiled packages.

But I guess in further versions the bugfix will be included.

Daniel Banck » bug

POST Data with .net Html Agility Pack

Bug in cherokee