Home > Fluid Dynamics Search Engine > Help > 1050

Filter Rules: Avoiding redundant Apache directory listings

The popular Apache web server will show the contents of directories, and will allow visitors to sort the contents using the query string N|M|S|D=A|D. See http://xav.com/i/ as an example - the "Name", "Last modified", "Size", and "Description" links allow the visitor to sort the list.

These query string options cause FDSE to index 9 different versions of the directory list (1 valid URL and 8 redundant URL's with sort parameters). A Filter Rule can be used to avoid the duplicate entries.

Note: most installs of FDSE will already contain an enabled copy of this Filter Rule, named "Skip extra Apache directory lists". If this rule is already present, you can just confirm that the "Enabled" property is checked and you will be set. Otherwise, go to "Admin Page" => "Filter Rules" => "Create New Rule" and create a new rule with the following parameters:

Parameter Value
Name: "Skip extra Apache directory lists"
Enabled: [x]
Action: (*) Deny
Analyze: (*) URL
Occurrences: 1
Logic: (*) Apply rule only if...
( ) Always apply rule, unless...
Strings: (empty)
Patterns:

After creating and/or enabling this filter rule, you will need to rebuild your search index to have the changes take effect.

In some rare cases, there will be valid URL's which end in "/?N=A" or some related pattern. These valid URL's would be excluded by this filter rule. If your site runs into this conflict, either disable this rule altogether, or create an "Always Allow" rule which applies to the specific set of valid URL's. The "Always Allow" rule will override the "Deny" action in this rule.

See also Filter Rules: Creating a new Filter Rule and Directory listings in the search results.


    "Filter Rules: Avoiding redundant Apache directory listings"
    http://www.xav.com/scripts/search/help/1050.html