Dealing with server timeouts and killed processes
The web server may kill any CGI process that uses too much time or too many resources. Each web server is configured with different threshold limits for time and CPU/memory resources.
How to detect a killed process
The server itself does not usually print any output to the screen when a process is killed. You will simply notice that the browser has stopped loading the page, and yet the page does not contain the complete output that you expected. Here is an example:
Note that with traditional CGI scripts, a page of output which stops printing may indicate that the process was killed by the server, or that the Perl script contained a runtime code error which caused the Perl interpreter to abort. The Fluid Dynamics Search Engine contains a global exception handler which traps all runtime exceptions and prints an error report when they occur. Because of this, pages of FDSE output which stop midway through without any error message are almost certainly due to a server kill.
Note also that servers decide which CGI processes to kill by sampling the set of all processes, instead of checking each process. This ensures that all runaway processes will eventually be stopped, while not requiring too much overhead for auditing. As a result, if you launch a resource-intensive CGI script several times, you may see it killed at different places in its execution, or it may succeed sometimes and fail other times. This is normal.
Also, note that FDSE contains a web crawler with is active on the network. While it is on the network, it may hang for extremely long periods of time. This will appear as a page which has stopped processing in the middle, but your browser should tell you that it is still listening for content from the server. This is usually communicated subtly through movement within the browser logo in the far upper right of your screen. The server kill happens when your browser is no longer listening for content.
As of FDSE version 2.0.0.0056, those using the admin page with Internet Explorer with Javascript enabled will see the error message "Warning: uncontrolled exit by Perl CGI script" when the server kills the process.
How to handle occasional killed processes
Try these ideas to handle occasional failures:
-
When a page fails, just reload it. Since FDSE uses a frames-based admin page, you will probably need to click in the content frame, and then bring up the context menu (by right-clicking in Windows) and choose "Reload Frame".
All operations in FDSE can be re-submitted without causing problems. For example, if you are creating a realm, and the "Create Realm" action is killed in the final step, you can simply re-submit the action as many times as necessary until it completes. There is no risk of accidentally creating dozens of realms with the same name.
-
You can give FDSE hints about how much time and how many resources it can use before it is killed. FDSE will try to monitor itself and complete and save its work before it reaches those limits. The General Settings "Timeout", "Network Timeout", "Time Interval Between Restarts", and "Crawler: Max Pages Per Batch" are all settings which control the workload of major tasks which are spanned across multiple CGI executions.
If you are experiencing a lot of killed processes, you should lower the timeouts and the pages-per-batch setting, and you should increase the time interval between restarts.
-
In addition to the workload settings, there are dozens of options in FDSE which can improve efficiency and speed. A full discussion of these settings can be found in the help file How to make indexing faster. Faster indexing and more efficient indexing go hand in hand.
-
Creating usage graphs can use a lot of memory when the log file is large. You may want to delete the log file regularly so that graphing processes are less likely to be killed.
How to handle frequent killed processes
On some servers, most or all of the tasks you attempt will fail because the server kills the process. Here are some ideas for handling this problem:
-
You should try to reduce the workload and resource usage using the ideas listed above.
-
Or, if reduced workload settings do not solve the problem, contact your web hosting provider. Plead with them to increase the CGI resource limits. Tell them:
"Increasing your limits by a factor of 5x or 10x will still protect your server from runaway processes, while allowing me to get my work done," and:
"While I can understand that you want to keep resource-intensive CGI scripts off of the server, my script is failing during rare admin tasks. I just need to periodically run certain resource-intensive admin tasks to build lean index files so that the search CGI processes can run as quickly as possible. This ensures that overall resource usage is minimized. Can you set higher CGI resource limits during certain off-peak times? Or, can you apply custom limits to my account or certain folders within my account?"
-
Or, you may install FDSE onto a separate server or your local computer. You can use this second copy of FDSE to build your index files and to do other resource-intensive tasks, and you can then FTP transfer the completed index files over to your main server.
Similarly, you can install FDSE onto a separate web hosting provider with more liberal resource limits, and you can use this as your dedicated search server, while still maintaining your main account on your main web hosting provider.
For more on this option, see Building index files remotely.
-
Or, you may want to not use FDSE at all, and instead use a remotely-hosted search solution like Google, Atomz, or Freefind. Remotely-hosted solutions do not run any code on your server and so CGI resource limits won't matter.
-
Or, you may want to try a different search engine. FDSE has been designed to be feature-rich at the cost of using more resources, especially for admin tasks. There are hundreds of other great search engines listed at CGI Resources. One of them may better fit your needs.
-
Or, you can switch web hosting providers to one with higher resource limits.
"Dealing with server timeouts and killed processes"
http://www.xav.com/scripts/search/help/1162.html