Screen Scraper that does Automated data entry.

2 May 2006 5:39 AM

swestenra

I am trying to build a screen scraper.
But not just a plain screen scraper, it must also automate the entry of
data.

Background:
We have a new intranet system that goes in to production soon. We'll
call it Grelb (not its real name).
On Grelb we need to add a few thousand users through its front end.
We will also need to maintain these users and update them, and so we
have another intranet app which I am developing.
My app maintains the users and their rights and many other values.
Grelb isn't it's only purpose.
My app uses VB.Net 2003 and SQL (containing the list of the thousands
of users).

I want to be able to press a button on my new app that opens up a
session of Grelb (in MSIE)
My app works through the SQL database and for each record it will click
the necessary buttons in Grelb, select the right option buttons, and
enter in the right text values.
And then click the submit button to create or modify a users account.
Repeat for a few thousand records.

I have looked up third party screen scrapers (SWEA, Kapow, WebInject,
TclWebtest, SnagIt, Mercury, Httpunit, OpenSta, Roboweb, WATIR) and
some Spam tools and some Robots, but none of these do the job.
I'm now trying it in VB.NET myself - which seems a lot simpler in
principal.

However I'm still trying to find a decent sample. Google, Google
groups and MS websites never quite go as far as the solution that I
need.

Has anyone developed something similar?
I've tried System.Net.WebRequest as well as System.Net.HTTPWebResponse.
My best bet seems to be System.Net.WebClient using DownloadData and
UploadData.

Problem 1) how to I display my destination web page in another IE
window that I can manipulate.

Problem 2) What to do with web pages that have Java Script in them.
"view source" returns
<head>
    <link rel="stylesheet" href="../standard.css" type="text/css">
    <link rel="stylesheet" href="../custom.css" type="text/css">
    <script language=javascript src=../../js/Standard.js></script>
    <script language=javascript src=../../js/Custom.js></script>
</head>
<h2 align=center>System Maintenance</h2>
<table>
    <tr><td>
      This is a secure area to allow the maintenance of users
    </td></tr>
</table>
[EOF]
which means I can't determine what the controls are called. I can't
give them focus or activate them.

Problem 3) a link brings up another IE window to add in further
details.
How can I reference these other IE windows?

I am wondering if I need to change from a web scraper to a control that
moves and activates the mouse, or instead uses sendkeys to Tab through
the controls.

Any help would be greatly appreciated.

I would have thought there'd be a niche out there for software like
this...

Thanks in advance
Simon Westenra
Wellington
New Zealand

3 May 2006 3:22 PM

alex_f_il

I have been used SWEA (http:\\webunittesting.com) on several very
compex screen scraping projects. Why it does't fit your requrements?