POSTing data with HttpWebRequest - problems connecting

akm

Newcomer
Joined
Dec 12, 2003
I'm making a program that attempts to POST form data to the All Music Guide site, get the server's HTML response, and then parse it to extract various bits of information. I think I understand what I have to do, but I'm having trouble getting it to work.

Basically, I use a HttpWebRequest to connect to the AMG site, POST the data, and then get the response stream and read in the response. However, I randomly get one of two errors - first, I sometimes get a WebException from HttpWebRequest's CheckFinalStatus() method, with message "The underlying connection was closed - an unexpected error occured on a receive".

I've searched the net for this particular exception, and the two solutions I've seen are modifying the KeepAlive property (setting it to false), and changing the .Net configuration file "machine.config" to not use the default proxy. However, neither of these suggestions seems to have helped. In fact, changing the machine.config file to not use the system default proxy settings makes the program fail 100% of the time (my IE connection settings do not have the box "Use a proxy server" ticked).

Another problem, which occurs less frequently, is that I sometimes get back a HTML page from the server looking something like this:
<HTML><HEAD><META HTTP-EQUIV="Refresh" CONTENT=0; URL=http://www.allmusic.com/cg/amg.dll></HEAD><BODY></BODY></HTML>

The URL is the DLL to which I'm sending the form data. So does this mean that I'm supposed to repost the data? I understand that on a browser this would attempt to immediately refresh to the URL given, but does this mean that the browser must repost the data? I'm not sure whether this is standard behaviour or whether there's something wrong (because I would imagine that the server would choose to wait until it gave me proper data, instead of making me redo my query).

Here's what my code looks like:
Code:
HttpWebRequest webRequest = null;

webRequest = (HttpWebRequest) WebRequest.Create (AMG_QUALIFIED_DLL); 

webRequest.KeepAlive = false;
webRequest.Timeout = 100000;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";

// Convert the postData into a byte array
// postData is an instance variable
byte[] requestBytes = (new System.Text.ASCIIEncoding()).GetBytes (postData);
webRequest.ContentLength = requestBytes.Length;

// Write the bytes to the request stream
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write (requestBytes, 0, requestBytes.Length);
requestStream.Close();

HttpWebResponse webResponse = null;
StreamReader sr = null;

try
{
    webResponse = (HttpWebResponse) webRequest.GetResponse(); 

    sr = new StreamReader (webResponse.GetResponseStream(), System.Text.Encoding.ASCII);
    response = sr.ReadToEnd ();
}

catch (WebException ex)
{
    Console.WriteLine (ex.Message + ex.StackTrace);
    throw new WebException (ex.Message, ex.InnerException, ex.Status, ex.Response);
}

finally
{
    if (sr != null)
        sr.Close(); 
    if (webResponse != null)
        webResponse.Close ();

    sr = null;
    webResponse = null;
}

Now, what really puzzles me is this - if I use the ActiveX WebBrowser control to POST the data, it seems to work fine 100% of the time! It would appear then that there is something wrong with what I'm doing, but I'm not sure what.

Here's the WebBrowser code (it's just a slightly modified version of the MS sample code) :

Code:
object oEmpty = "";
string cPostData = "p=amg&sql="+artistNameBox.Text+"&opt1=1";
object vHeaders = "Content-Type: application/x-www-form-urlencoded" + "\n" + "\r";
object vPost = System.Text.ASCIIEncoding.ASCII.GetBytes(cPostData);

this.axWebBrowser1.Navigate ("http://www.allmusic.com/cg/amg.dll", ref oEmpty, ref oEmpty, ref vPost, ref vHeaders);

I though that perhaps this had to do with the UserAgent property of the request, and so I tried setting it to the string for IE 6 (I can't remember the string, something like "Mozilla 4.0; compatible IE6.0"?). However, that didn't seem to change things either.

Any help would be much appreciated!
 

karlhaak

Newcomer
Joined
Feb 2, 2004
connection closed on webrequest

Thank you very much for your post here.
The option with keepAlive=false works excellent for me. I was experiencing the following behavior.

I was writing a windows service that would connect to a website, post some data to get a response. Everything worked fine when I ran the component as a console Application in Debug mode on my machine . Once installed as a service every second request failed.

I had to fight with an unvalid certificate for ssl, username, password and a proxyserver which gave me plenty of options to look for.

I finally tried the keepAlive option and everything works fine. I still do not understand the influence of this parameter on the failure of the connection every second time and would be interested in any ideas.

The environment I am working in is Windows 2000 with .NET 1.0.

I created an abstract of my solution as a coding example. Hope this will safe some time for somebody else.

Karl

Code:
Imports System.IO
Imports System.Text
Imports System.Threading
Imports System.Net
Imports System.Text.Encoding
Imports System.Security.Cryptography.X509Certificates

Public Class MyCertificateValidation
    Implements System.Net.ICertificatePolicy

    'This class handles problems with certificates if ssl (https) is used

    Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
    ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
       As Boolean Implements ICertificatePolicy.CheckValidationResult
        Return True ' Accept all certificates
    End Function
End Class

Public Class RequestState
    Public request As WebRequest = Nothing
    Public requestDocument As String
End Class

Public Class HTMLRequest

    'Handler to synchronise with main thread again
    Public allDone As New ManualResetEvent(False)

    Private _requestDocument As String = "Text to send"
    Private _responseDocument As String
    Private _web As Net.HttpWebRequest

    Private Sub ReadCallback(ByVal asynchronousResult As IAsyncResult)
        'Get the information to pass on from the Async Object
        Dim myRequestState As RequestState = CType(asynchronousResult.AsyncState, RequestState)
        Dim myWebRequest2 As WebRequest = myRequestState.request
        Dim requestDocument As String = myRequestState.requestDocument

        'Get the stream to write the request to
        Dim streamResponse As Stream = myWebRequest2.EndGetRequestStream(asynchronousResult)
        Dim bytArray() As Byte = System.Text.Encoding.ASCII.GetBytes(requestDocument)
        streamResponse.Write(bytArray, 0, requestDocument.Length)
        streamResponse.Close()

        'Tell the main thread that writing to stream is complete
        allDone.Set()
    End Sub

    Public Sub retrieve()
        Dim bytOutArray() As Byte
        Dim objWebResponse As Net.WebResponse = _web.GetResponse
        Dim objStream As IO.StreamReader
        objStream = New IO.StreamReader(objWebResponse.GetResponseStream(), System.Text.ASCIIEncoding.ASCII)
        _responseDocument = objStream.ReadToEnd()
        'close all streams
        objWebResponse.Close()
        objStream.Close()
    End Sub

    Public Sub send()
        'Prepare Object to pass into asynchron call
        Dim myRequestState As New RequestState
        myRequestState.request = _web
        myRequestState.requestDocument = _requestDocument
        Dim r As IAsyncResult = CType(_web.BeginGetRequestStream(AddressOf ReadCallback, myRequestState), IAsyncResult)

        'Wait till asychrone is done ## allDone.Set() ##
        allDone.WaitOne()
    End Sub

    Public Sub getRequest()

        'Configure connection

        'in case of problem with certificate
        ServicePointManager.CertificatePolicy = New MyCertificateValidation

        'Create request object
        _web = CType(WebRequest.Create("https://www.somedomain.com"), HttpWebRequest)
        With _web
            .ContentType = "application/x-www-form-urlencoded"
            .Method = "POST"
            .KeepAlive = False
            .ContentLength = _requestDocument.Length
            .Timeout = 10000

            'in case of proxyserver
            .Proxy = New WebProxy("xxx.proxyserver.com", 8080)

            'in case of required password
            Dim cache As CredentialCache = New CredentialCache
            cache.Add(New Uri("https://www.somedomain.com/"), "Basic", New NetworkCredential("Username", "password"))
            .Credentials = cache
        End With

        send() 'send the request
        retrieve() 'get the data
    End Sub
End Class
 
Last edited:

aric

Newcomer
Joined
Jul 29, 2004
I have hit the same problem...

I am trying to implement a financial gateway system, which uses WSE 1.0 SP1, and the sample code provided makes an SSL connection to their server. So, I have to find a way to override the System.Net.ICertificatePolicy so as to always approve SSL certs.

I am running ASP.NET (VB) .NET version 1.1, and am having a hard time getting your code to work, and was wondering if you had any suggestions.

I put your class above my class (in a code-behind page which inherits the page) thusly:

Code:
Public Class MyCertificateValidation
    Implements System.Net.ICertificatePolicy

    'This class handles problems with certificates if ssl (https) is used

    Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
    ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
       As Boolean Implements ICertificatePolicy.CheckValidationResult
        Return True ' Accept all certificates
    End Function
End Class

Public Class thepage

    ' Inherit the page
    Inherits Page
.
.
.

Everything looks good, but I get this error:

Code:
Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.

Compiler Error Message: BC30149: 'MyCertificateValidation' must implement 'Overridable Function CheckValidationResult(srvPoint As ServicePoint, certificate As Security.Cryptography.X509Certificates.X509Certificate, request As WebRequest, certificateProblem As Integer) As Boolean' for interface 'System.Net.ICertificatePolicy'.

Source Error:

Line 26: 
Line 27: Public Class MyCertificateValidation
Line 28:     Implements System.Net.ICertificatePolicy
Line 29: 
Line 30:     'This class handles problems with certificates if ssl (https) is used

However, it clearly DOES overright it... and I am stumped. :confused:

Any ideas anyone?

Thanks in advance.
 

karlhaak

Newcomer
Joined
Feb 2, 2004
Quick suggestions for: Compiler Error Message: BC30149

Hello Aric,

I cannot see any problem at first glance. Here are some quick things I would like to check.

1) If you programm with visual studio, it will mark the class as invalid if not all declared interfaces have been implemented. Can you check that.

2) Put the class in an extra file.

3) check if the namespace
Imports System.Security.Cryptography.X509Certificates
has been imported.

4) check if the namespace
Imports System.Net
has been imported.


hope one of these will help

Karl
 
Top Bottom