[Python-au] Webscraping

Roger Barnes roger at mindsocket.com.au
Sat Apr 23 12:07:29 UTC 2011


Mechanize combined with Beautiful Soup are often a good combo. YMMV

- Roger
On Apr 23, 2011 9:51 PM, <trideceth12 at gawab.com> wrote:
> Hi all,
>
> Can anyone recommend me a python package for handling webscraping
> operations. I need to be able to log-in to an https site and crawl from
> there.
>
> I have been trying to use HtmlUnit for java and have seen some people
> using HtmlUnit and Jython, but so far HtmlUnit seems a bit flaky -
> retaining logged-in status on some sites, not on others.
>
> Is this really so hard???? I'm sure this must be a common operation.
>
> Thanks in advance,
> Jake
>
>
>
> _______________________________________________
> python-au maillist - python-au at starship.python.net
> http://starship.python.net/mailman/listinfo/python-au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://starship.python.net/pipermail/python-au/attachments/20110423/60655e5b/attachment.htm>


More information about the python-au mailing list