Home > Fluid Dynamics Search Engine > Help > 1065

Building index files remotely

The index files are portable; they can be created on a computer separate from your search machine.

This is recommended for busy sites, since the index-update process is resource intensive and may interfere with searching.

Follow these steps:

  1. First, install FDSE onto a second web server. You can use your local computer if it has been configured to run as a Perl CGI web server.

  2. Second, copy all files and folders from the /search/searchdata/ folder on your main web server over to your second copy of FDSE. Use the "binary" transfer type if using FTP.

    Then, from your second copy of FDSE, go to the admin page and perform any admin tasks you want, such as rebuilding your search indexes.

    Note: if you allow visitor-added pages, you must configure all submissions to "require administrator approval". Then, be sure to periodically transfer the "*.need_approval" files down from your main search engine to your secondary search engine. Then, from your secondary search engine, walk through the approval process and upload the completed index files.

  3. Finally, copy the updated files back to your main web server. Use "binary" format if transferring with FTP. The files that are likely to be updated are:

    • searchdata/index_file* (or whatever your index files have been named)

    • searchdata/custom_metadata*

    • searchdata/filter_rules.txt

    • searchdata/search.pending.txt

    • searchdata/search.realms.txt

    • searchdata/settings.pl

The FDSE license allow you to use the same registration key for one public search instance and any number of non-public secondary instances which support it.

Using a free secondary web server

There are several companies that offer free web hosting with CGI privileges, such as netfirms.com, hypermart.net, and virtualave.com. Unfortunately, these companies tend to not allow CGI scripts to make socket connections, which prevents the FDSE web crawler from connecting to the network. In addition, these providers tend to have very strict CGI resource limits. Free web hosting providers are not good candidates for hosting a secondary instance of FDSE.

A better solution is to use low-cost web hosting providers, like pair.com, which offer full CGI with sockets privileges for around $20/month.

Turning your local computer into a web server

Setting up your local computer as a web server is desirable for many reasons. The primary benefit is that you can devote all of your computing resources to resource intensive processes, without causing trouble on your live web server. A local web server also makes it easy to run development and test copies of all your web sites.

Turning your local computer into a web server can take between two hours and two weeks, depending on your experience and on the type of local computer you have. Note that setting up your local computer as a web server makes it easier for outsiders to hack into your local system, particularly if you are directly connected to the Internet. The risk can be greatly reduced, but not completely eliminated, by configuring the server properly and regularly applying the latest patches as they become available. Because of the complexity, liability, and required ongoing maintenance, Fluid Dynamics does not provide assistance with turning your local computer into a web server. There are probably some good tutorials out there that can help with it.

    "Building index files remotely"