Home > Fluid Dynamics Search Engine > Help > 1125

Filter Rules: Using the "index, nofollow" filter rule

The filter rule "Index document, but do not follow links", also known as "index,nofollow", is designed to mimic the behavior of the "index,nofollow" robots meta tag. A document affected by this rule will be indexed as normal, but any links found in the document will not be followed.

An example of a user scenario for this rule would be indexing the main page of a message board system of a site. You might choose to index the contents of main page, which includes the subject lines of all messages, but you choose to not dig deeper and index all individual messages which are linked from that main page. In this example, the message board might reside on a remote system and so you are unable to add a normal "meta robots index,nofollow" tag to its header, and so creating an "index,nofollow" filter rule is the next best solution.

To create an "index,nofollow" filter rule, follow these steps:

  1. Log in to the "Admin Page".

  2. Choose the "Filter Rules" link from the navigation menu. From that page, choose Filter Rules - Create New Rule.

  3. You will be taken to the Create or Update Rule page. Enter these values:

    Name: My Rule
    Enabled: [x] (checked)
    Action: (*) Index document, but do not follow links
    Analyze: (*) URL
    Minimum Occurrences: 1
    (*) Apply rule only...
    Strings: (enter URL's or hostnames here)
    Scope: (*) Apply to all realms

  4. Click the "Save Data" button to save your new rule.

  5. Test the rule by going to "Admin Page" and adding a URL in the "Add New URL" form. Make sure the original URL has been listed in the filter rule "strings" section that was just created. After indexing, FDSE should respond with "No embedded links were found during this crawl session".

Note that links are only followed for web-crawler type realms in any case, not file-system crawler realms. The latter realms will not behave differently based on this type of rule.

History: this type of filter rule was added with FDSE version

    "Filter Rules: Using the "index, nofollow" filter rule"