ActivePerl Documentation
Table of Contents
Getting Started
Welcome To ActivePerl Release Notes Readme ActivePerl Change Log Install Notes Linux Solaris Windows ActivePerl Components Overview PPM Windows Specifics OLE Browser PerlScript Perl for ISAPI PerlEZ ActivePerl FAQ Introduction Availability & Install Using PPM Docs & Support Windows Specifics Perl for ISAPI Windows 9X/NT/2000 Quirks Web Server Config Web programming Programming Modules & Samples Embedding & Extending Using OLE with Perl Windows Scripting Active Server Pages Windows Script Host Windows Script ComponentsCore Perl Documentation
perl perlfaq perltoc perlbook perlsyn perldata perlop perlreftut perldsc perllol perllexwarn perldebug perlrun perlfunc perlopentut perlvar perlsub perlmod perlpod perlstyle perlmodlib perlmodinstall perltrap perlport perlsec perlref perlre perlform perllocale perlunicode perlboot perltoot perltootc perlobj perlbot perltie perlipc perlnumber perlfork perlthrtut perldiag perlfaq1 perlfaq2 perlfaq3 perlfaq4 perlfaq5 perlfaq6 perlfaq7 perlfaq8 perlfaq9 perlcompile perlembed perlxstut perlxs perlguts perlcall perlfilter perldbmfilter perlapi perlintern perlapio perltodo perlhack perlhist perldelta perl5005delta perl5004delta perlamiga perlcygwin perldos perlhpux perlmachten perlos2 perlos390 perlvms perlwin32Pragmas
attributes attrs autouse base blib bytes charnames constant diagnostics fields filetest integer less lib locale lwpcook open ops overload perllocal re sigtrap strict subs utf8 vars warningsLibraries
ActivePerl DocTools TOC RDF AnyDBM_File Archive Tar AutoLoader AutoSplit B Asmdata Assembler Bblock Bytecode C CC Debug Deparse Disassembler Lint Showlex Stackobj Terse Xref Benchmark Bundle LWP ByteLoader Carp Heavy CGI Apache Carp Cookie Fast Pretty Push Switch Class Struct Compress Zlib Config CPAN FirstTime Nox Cwd Data Dumper DB Devel DProf Peek SelfStubber Digest HMAC HMAC_MD5 HMAC_SHA1 MD2 MD5 SHA1 DirHandle Dumpvalue DynaLoader English Env Errno Exporter Heavy ExtUtils Command Embed Install Installed Liblist MakeMaker Manifest Miniperl Mkbootstrap Mksymlists MM_Cygwin MM_OS2 MM_Unix MM_VMS MM_Win32 Packlist testlib Fatal Fcntl File Basename CheckTree Compare Copy CounterFile DosGlob Find Glob Listing Path Spec Functions Mac OS2 Unix VMS Win32 stat FileCache FileHandle FindBin Font AFM Getopt Long Std HTML AsSubs Element Entities Filter Form FormatPS Formatter FormatText HeadParser LinkExtor Parse Parser TokeParser TreeBuilder HTTP Cookies Daemon Date Headers Util Message Negotiate Request Common Response Status I18N Collate IO Dir File Handle Pipe Poll Seekable Select Socket INET UNIX IPC Msg Open2 Open3 Semaphore SysV LWP Debug MediaTypes MemberMixin Protocol RobotUA Simple UserAgent Math BigFloat BigInt Complex Trig MD5 MIME Base64 QuotedPrint NDBM_File Net Cmd Config Domain DummyInetd FTP hostent libnetFAQ netent Netrc NNTP PH Ping POP3 protoent servent SMTP SNPP Time O ODBM_File Opcode Pod Checker Find Html InputObjects Man Parser ParseUtils Plainer Select Text Color Termcap Usage POSIX PPM SOAPClient SOAPServer Safe SDBM_File Search Dict SelectSaver SelfLoader SHA Shell SOAP Defs Envelope EnvelopeMaker GenericHashSerializer GenericInputStream GenericScalarSerializer Lite OutputStream Packager Parser Transport HTTP Apache CGI Client Server LOCAL MAILTO POP3 TCP TypeMapper Socket Symbol Sys Hostname Syslog Term ANSIColor Cap Complete ReadLine Test Harness Text Abbrev ParseWords Soundex Tabs Wrap Thread Queue Semaphore Signal Specific Tie Array Handle Hash RefHash Scalar SubstrHash Time gmtime Local localtime tm UDDI Lite UNIVERSAL URI data Escape file Heuristic ldap URL WithBase User grent pwent Win32 AuthenticateUser ChangeNotify Clipboard Console Event EventLog File FileSecurity Internet IPC Mutex NetAdmin NetResource ODBC OLE Const Enum NEWS NLS TPJ Variant PerfLib Pipe Process Registry Semaphore Service Sound TieRegistry Win32API File Net Registry WWW RobotRules AnyDBM_File XML Element Parser Expat PPD PPMConfig ValidatingElement XSLoader|
LWP::RobotUA - A class for Web Robots |
NAME
LWP::RobotUA - A class for Web Robots
SUPPORTED PLATFORMS
- Linux
- Solaris
- Windows
SYNOPSIS
require LWP::RobotUA; $ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com'; $ua->delay(10); # be very nice, go slowly ... # just use it just like a normal LWP::UserAgent $res = $ua->request($req);
DESCRIPTION
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently.
But, before you consider writing a robot take a look at <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.
When you use a LWP::RobotUA as your user agent, then you do not really have to think about these things yourself. Just send requests as you do when you are using a normal LWP::UserAgent and this special agent will make sure you are nice.
METHODS
The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. In addition the following methods are provided:
- $ua = LWP::RobotUA->new($agent_name, $from, [$rules])
-
Your robot's name and the mail address of the human responsible for
the robot (i.e. you) are required by the constructor.
Optionally it allows you to specify the WWW::RobotRules object to use.
- $ua->
delay([$minutes])
- Set the minimum delay between requests to the same server. The default is 1 minute.
- $ua->
use_sleep([$boolean])
-
Get/set a value indicating whether the UA should
sleep()if requests arrive too fast (before $ua->delay minutes has passed). The default is TRUE. If this value is FALSE then an internal SERVICE_UNAVAILABLE response will be generated. It will have an Retry-After header that indicates when it is OK to send another request to this server. - $ua->
rules([$rules])
- Set/get which WWW::RobotRules object to use.
- $ua->
no_visits($netloc)
-
Returns the number of documents fetched from this server host. Yes I
know, this method should probably have been named
num_visits()or something like that. :-( - $ua->
host_wait($netloc)
- Returns the number of seconds (from now) you must wait before you can make a new request to this host.
- $ua->as_string
- Returns a string that describes the state of the UA. Mainly useful for debugging.
SEE ALSO
the LWP::UserAgent manpage, the WWW::RobotRules manpage
COPYRIGHT
Copyright 1996-2000 Gisle Aas.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
|
LWP::RobotUA - A class for Web Robots |