Downloading multiple WebPages and saving to text file help needed!

14 Jun 2006 4:57 AM

Hexman

Code below ----

I'm trying to save some specific web pages to disk as text files. I searched the Internet and found a basic example which I changed to fit my needs.
I tested it out first on a single URL and it worked fine. Now when I incorporate an array of URL's, it fails to work. The first "responseFromServer"
properly retrieves and displays "http://finance.yahoo.com" (or any other valid URL for the first webrequest.create), but when I use the urls from the
array in the for/next structure, I never get a responseFromServer. My guess is that there must be some kind of initialization to the WebRequest,
Stream, or WebResponse.

A new command for me is "reader.ReadToEnd". In one fell swoop it reads the entire webpage. Very cool. Now I also want to write the entire
"responseFromServer" to disk as a text file. I've seen commands like "File.WriteAllText" and it seems to work well for what I need. Should I look
into any other commands to save the "responseFromServer" to disk as a text file?

Thanks,

Hexman

--------------------------------------------------------------------------------------------
Imports System
Imports System.IO
Imports System.Net
Imports System.Net.WebClient
Imports System.Windows.Forms
Public Class Form1

    Private Sub btnProcess_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnProcess.Click
        Dim x As Integer
        Dim aryUrls(15) As String
        aryUrls(0) = "http://finance.yahoo.com/q/bc?s=DOW&t=1d"
        aryUrls(1) = "http://finance.yahoo.com/q/bc?s=IBM&t=6m"
        aryUrls(2) = "http://finance.yahoo.com/q/bc?s=MSFT&t=6m"
        aryUrls(3) = "http://biz.yahoo.com/top.html"

        Dim request As WebRequest = WebRequest.Create("http://finance.yahoo.com/")
        Dim response As WebResponse = request.GetResponse()
        Dim dataStream As Stream = response.GetResponseStream()
        Dim reader As New StreamReader(dataStream)
        Dim responseFromServer As String = reader.ReadToEnd()

        MsgBox(responseFromServer) <=== Shows finance.yahoo.com

        For x = 0 To 3
            request = WebRequest.Create(aryUrls(x))
            dataStream = response.GetResponseStream()
            responseFromServer = reader.ReadToEnd()
            MsgBox(responseFromServer) <=== Shows nothing
        Next

        reader.Close()
        response.Close()
    End Sub

14 Jun 2006 5:40 AM

Cor Ligthert [MVP]

Hexman,

To check this kind of routines is a hell of a job and yours seems to me
incomplete so even not to debug..
It looks like the sample on our webpages but even than I find that always
painfull to debug.

http://www.vb-tips.com/default.aspx?ID=541adf13-d9c0-435c-893f-56dbb63fdf1c

Be aware however that if there are frames used on a webpage, you get with
this kind of routines only the MainFrame.

That was the first that came up into my mind.

Cor

Show quoteHide quote

"Hexman" <Hex***@binary.com> schreef in bericht
news:nk4v82lbk42pktoq9l53lkf2vpfrftkkl7@4ax.com...
> Code below ----
>
> I'm trying to save some specific web pages to disk as text files. I
> searched the Internet and found a basic example which I changed to fit my
> needs.
> I tested it out first on a single URL and it worked fine. Now when I
> incorporate an array of URL's, it fails to work. The first
> "responseFromServer"
> properly retrieves and displays "http://finance.yahoo.com" (or any other
> valid URL for the first webrequest.create), but when I use the urls from
> the
> array in the for/next structure, I never get a responseFromServer. My
> guess is that there must be some kind of initialization to the WebRequest,
> Stream, or WebResponse.
>
> A new command for me is "reader.ReadToEnd". In one fell swoop it reads
> the entire webpage. Very cool. Now I also want to write the entire
> "responseFromServer" to disk as a text file. I've seen commands like
> "File.WriteAllText" and it seems to work well for what I need. Should I
> look
> into any other commands to save the "responseFromServer" to disk as a text
> file?
>
> Thanks,
>
> Hexman
>
> --------------------------------------------------------------------------------------------
> Imports System
> Imports System.IO
> Imports System.Net
> Imports System.Net.WebClient
> Imports System.Windows.Forms
> Public Class Form1
>
>    Private Sub btnProcess_Click(ByVal sender As System.Object, ByVal e As
> System.EventArgs) Handles btnProcess.Click
>        Dim x As Integer
>        Dim aryUrls(15) As String
>        aryUrls(0) = "http://finance.yahoo.com/q/bc?s=DOW&t=1d"
>        aryUrls(1) = "http://finance.yahoo.com/q/bc?s=IBM&t=6m"
>        aryUrls(2) = "http://finance.yahoo.com/q/bc?s=MSFT&t=6m"
>        aryUrls(3) = "http://biz.yahoo.com/top.html"
>
>        Dim request As WebRequest =
> WebRequest.Create("http://finance.yahoo.com/")
>        Dim response As WebResponse = request.GetResponse()
>        Dim dataStream As Stream = response.GetResponseStream()
>        Dim reader As New StreamReader(dataStream)
>        Dim responseFromServer As String = reader.ReadToEnd()
>
>        MsgBox(responseFromServer) <=== Shows finance.yahoo.com
>
>        For x = 0 To 3
>            request = WebRequest.Create(aryUrls(x))
>            dataStream = response.GetResponseStream()
>            responseFromServer = reader.ReadToEnd()
>            MsgBox(responseFromServer) <=== Shows nothing
>        Next
>
>        reader.Close()
>        response.Close()
>    End Sub