Search engine script » file

CAN SEARCH ENGINE CRAWLERS STILL READ A DOMAINS ROBOTS.TXT FILE IF ITS PERMISSION IS SET TO 600?

admin — Tue, 26 Jul 2011 06:11:15 +0000

Question by Omnis:
Can search engine crawlers still read a domains robots.txt file if its permission is set to 600?
The domain is running on a linux server, and what I am trying to accomplish is have a robots.txt file that search engine crawlers can read but the any user cannot access/view via the browser.
The whole point of using a robots.txt file is to hide folders/files from ending up in search results in google, yahoo, etc, which in turn the public can access. Even though it hides folders/files from search engine crawlers, the actual robots.txt file is still accessible through http://www.domain.com/robots.txt, which displays all the folders/files I am trying to hide, unless you can place robots.txt file in the actual directory your trying to hide? I know you can password directories and thats what I intend to do as well.

——————————————

Answer by memetrader
No they can’t.

You could however use something like RewriteCond to detect the user-agent string Mozilla and then rewrite robots.txt to some other file. You would have to exclude the two or three bots that include Mozilla in the user-agent string though. Simple enough but why would you want to do this?

——————————————
Know better? Leave your own answer in the comments!

PHP SCRIPT FOR MAKING A FILE SEARCH ENGINE?

admin — Mon, 29 Nov 2010 17:16:02 +0000

Question by Plezops:
PHP Script for making a file search engine?
I need a PHP script that can search for a specific file (with a box to type in criteria) within a certain web directory (just one directory) on my server. Are there any FREE or cheap search engines that i can use and customize to fit my needs?

——————————————

Answer by Mardix
I would advocate you to create a table in a database that might have the following fields :
id (unique identifier, you might use it to update data),
url (the location of the file on your server),
title (to identify it),
description (to describe what is in the file),
keywords (words you might use to search the file)
date (date of insertion).

And when you want to search for a file, the search will be made in the database. You will be able to see the details such as description etc.
And when you click on it, you will be able to go to the file you want by using the field URL that is in the DB.

It will be faster this way. All you have to do in the DB is inserting new file, you do not have to make your script do too much for nothing.

——————————————
Give your own answer to this question below!

WORD FREQUENCY PHP COUNT ON TEXT FILE?

admin — Sun, 08 Aug 2010 05:34:29 +0000

Hi There,

I’m running a search engine for media files and I am trying to script a box with the 50 most popular search terms. My search engine currently has a text file which the site writes to each time someone searches a term. So all I need is a code to count the frequency of the terms in the text file and the 50 most popular need to be written (automatically) into the homepage of my site. Any help?

I’M USING A SCRIPT IN MY .HTACCESS FILE WHICH POINTS ALL LINKS FROM MY “NON-WWW” SITE TO MY WWW SITE.

admin — Fri, 30 Jul 2010 06:13:05 +0000

I’m using:

# Start non-www page protection #

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.mysite\.com [NC]
RewriteRule ^(.*)$ http://www.mysite.com/$1 [L,R=301]

# End non-www page protection #

Is this the correct format, and will all links now point just to the www version just from the time that I do this and upload the new .htaccess file, or will all links (and the page rank boost associated with it) become retrospective? Will I see a boost in rank immediately?

I’ve also heard that the search engines see a www.mysite.com version and a www.mysite.com/ version. Will my script take care of this as well, or do I have to add something else? (I cannot see whether the script does this or not as I am not a coder!)

For .co.uk domains do I just alternative .co.uk for .com in each case?

HOW CAN I CREATE A SEARCH ENGINE FOR MY HTML FILE?

admin — Thu, 03 Jun 2010 06:42:33 +0000

it is not linked to the internet

PHP SCRIPT FOR MAKING A FILE SEARCH ENGINE?

admin — Tue, 02 Mar 2010 19:31:00 +0000

I need a PHP script that can search for a specific file (with a box to type in criteria) within a certain web directory (just one directory) on my server. Are there any FREE or cheap search engines that i can use and customize to fit my needs?