I came across need the other day where I had to scrape a page. The issue was that if you are not logged in then you get different content to if you were logged in. It was a relatively easy solution but I came across some code that really stood out at stackoverflow.
The bulk of it a nice simple class to encapsulate the cookies and authentication credentials in a class as follows:
public class CookieAwareWebClient : WebClient
{
public CookieAwareWebClient()
{
CookieContainer = new CookieContainer();
}
public CookieContainer CookieContainer { get; private set; }
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
Thus once you provide login details you can then easily call on the WebClient method as illustrated in sample code (also from the stackoverflow site):
using (var client = new CookieAwareWebClient())
{
var values = new NameValueCollection
{
{ "username", "john" },
{ "password", "secret" },
};
client.UploadValues("http://domain.loc/logon.aspx", values);
// If the previous call succeeded we now have a valid authentication cookie
// so we could download the protected page
string result = client.DownloadString(“http://domain.loc/testpage.aspx”);
}
Again this is not my code – I just liked it’s elegance in solving a simple problem.
