Website page scraping through HTTP Post

, ,
1

Hi All,

We have a requirement to develop a interface to log into a client website and pass inputs in a particular page and get the output from it. Please find the process below.

  1. Login into a website using User ID and Password.
  2. navigate to a particular page.
  3. Provide inputs in that page and submit.
  4. Confirm the provided inputs.
  5. Read a particular data from the output html page.

Note: website is not implemented for m/c to m/c communication, but requirement is to implement an interface using webMethods.
We have enabled the Cookies handler in the extended settings and tried to do a HTTP POST to the website passing all the required inputs. We are always being redirected to the login page.

Please help us in providing a solution to this.

Thanks.

2

I don’t know of any current webMethods products that support website screen scraping. That integration approach was always very unreliable since the most minor change in the source website will cause it to fail.

Mark

3

Hi Mark,

Thanks for the response.
We are aware that this approach is unreliable, but this is the only was for the business to full fill the requirements at present, and i got a suggestion that if we enable the cookies handling we can achieve the website scraping. So can you plese help me in this.

Thanks,
Ram.

4

I don’t think that would help, but feel free to try it. I do not believe webMethods IS is the right tool for this job.