[triangle-zpug] Python spider

Tom Bryan tbryan at python.net
Sat Jul 10 19:47:15 CEST 2004


On Wednesday 07 July 2004 04:31 pm, David Chandek-Stark wrote:
> Has anyone written a spider in Python -- or know of one -- that would be
> useful in "dumping" a Zope site to the file system? I'm working with
> Wget now, and it may be sufficient for my needs, but I'd like to know
> that I could fall back on a customized Python solution, if necessary.

The Python distribution comes with (or, it used to come with) a utility called 
webchecker.  If you're on a UNIX system, and you built Python from source, 
you'll find it under the Tools directory.  For example, 

/usr/local/python-2.1/Tools/webchecker/tktools.py
/usr/local/python-2.1/Tools/webchecker/README
/usr/local/python-2.1/Tools/webchecker/wcgui.py
/usr/local/python-2.1/Tools/webchecker/wcmac.py
/usr/local/python-2.1/Tools/webchecker/webchecker.py
/usr/local/python-2.1/Tools/webchecker/websucker.py
/usr/local/python-2.1/Tools/webchecker/wsgui.py

The websucker.py script can be used to spider a site/create a local copy of a 
site.  It has failed on some sites, but it has generally suited my needs when 
I needed something like this.

---Tom




More information about the triangle-zpug mailing list