Home > Fluid Dynamics Search Engine > Help > 1090

Running as a "meta search" to display results of other search engines

Some search engines perform a search "under the hood" to a major engine, then parse the results and display them as if they came from the local site. These are known as "meta search engines". They can quickly search large data sets, since they rely on the dedicated servers and expertise of the major search engine that's running behind the scenes.

FDSE does not do this. FDSE allows you to build your own small custom search index, and then it queries that index. It does not have the ability to query other search engines.

There is a way to create keyword-links to major search engines for your visitors to follow; see: Searching secondary search engines. Those links allow the FDSE visitor to click-through to other major search engine, using an "above the hood" link.

Will the "meta search" feature be added?

I have decided not to take FDSE in this direction because most major search engines have policies against being spidered by meta scripts. They don't seem to enforce those policies, but it is still a questionable business practice to make unauthorized, automated queries against them.

Not all Perl developers feel this way. You can find many meta scripts at CGI Resources under the Remote Search category. Before investing time or money into a meta search script, make sure it has authorized, stable and affordable data feeds. At least make sure that they're stable.

Google recently released an API that allows programs to perform automated searches. I wrote to Google and asked whether I could create a meta search engine based on their API. On 2002-08-14, Google responded with a polite "no", reminding me that their API cannot be used for commercial applications, nor for applications which compete with Google. There you have it. If major search engines ever do offer an authorized API for queries then I might customize FDSE to use them; until then I will not bother with it.


    "Running as a "meta search" to display results of other search engines"
    http://www.xav.com/scripts/search/help/1090.html