POST Data with .net Html Agility Pack

I’m using the Html Agility Pack for getting and parsing HTML pages. It offers many possibilities, including XPath selectors.

Today I had a problem with posting data to a webpage. It’s possible to call the HtmlWeb Load function with a second “POST” parameter, but it offers no easy way to add values to be posted.

After a while of searching I came across this solution: add this two functions to the HtmlWeb class. (You can get the source in download section, too)

public HtmlDocument SubmitFormValues (NameValueCollection fv, string url)
{
    // Attach a temporary delegate to handle attaching
    // the post back data
    PreRequestHandler handler = delegate(HttpWebRequest request) {
        string payload = this.AssemblePostPayload (fv);
            byte[] buff = Encoding.ASCII.GetBytes (payload.ToCharArray ());
            request.ContentLength = buff.Length;
            request.ContentType = "application/x-www-form-urlencoded";
            System.IO.Stream reqStream = request.GetRequestStream ();
            reqStream.Write (buff, 0, buff.Length);
            return true;
    }
    this.PreRequest += handler;
    HtmlDocument doc = this.Load (url, "POST");
    this.PreRequest -= handler;
    return doc;
}

private string AssemblePostPayload (NameValueCollection fv)
{
    StringBuilder sb = new StringBuilder ();
    foreach (String key in fv.AllKeys) {
        sb.Append ("&" + Uri.EscapeDataString(key) + "=" + Uri.EscapeDataString(fv.Get(key)));
    }
    return sb.ToString ().Substring (1);
}

Source: http://htmlagilitypack.codeplex.com/Thread/View.aspx?ThreadId=14255

Now posting is easy. Example:

NameValueCollection postData = new NameValueCollection (1);
postData.Add ("name", "value");
doc = this.hw.SubmitFormValues (postData, url);

(hw is an HtmlWeb object and doc an HtmlDocument object)

If you need to use cookies, too, you need to modify the HtmlWeb a little more, because there is a little bug. First add a new private variable:

private CookieContainer _ck = new CookieContainer();

Then find this section:

if (_useCookies)
{
        req.CookieContainer = new CookieContainer();
}

And change it into:

if (_useCookies)
{
        req.CookieContainer = this._ck;
}

Now you just have to activate the usage of cookies in your HtmlWeb object:

this.hw = new HtmlWeb ();
this.hw.UseCookies = true;

That’s it! Enjoy easy form posting with a full cookie jar (: