Home > Fluid Dynamics Search Engine > Help > 1101

Using the web crawler through a proxy server

If there is a proxy server between your FDSE server and the web server you wish to index, you can configure FDSE to route through that proxy.

To do this, open the library file "common_admin.pl" in the "searchmods" folder. Find the Crawler_new subroutine. It looks like this:

sub Crawler_new {
	my $self = {
		'b_use_proxy' => 0,
		'proxy_addr' => 'proxy',
		'proxy_port' => 80,
		};

Configure the settings as follows, where the items in bold have been customized to your network:

sub Crawler_new {
	my $self = {
		'b_use_proxy' => 1,
		'proxy_addr' => 'my.proxy.address.here',
		'proxy_port' => 8000,
		};

The FDSE proxy setting is an all-or-nothing setting. When the b_use_proxy bit is set, all requests will route through the proxy. There is no option to only retreive certain sites through the proxy.

History: the proxy support code has been present in almost all versions. Prior to version 2.0.0.0032, the proxy code was located in the "crawler.pl" library, before that library was merged with "common_admin.pl".


    "Using the web crawler through a proxy server"
    http://www.xav.com/scripts/search/help/1101.html