Zen Internet Support Forum

I am think I need to put a robots.txt file in the 'top level' of my directory structure.  So I am placing it in ftp.[mysite].myzen.co.uk

Is that correct?  Or should it be in ftp.[mysite].myzen.co.uk/public_html  where I all the html pages for my site reside?

Do I need to [edit] exclude all those top level folders (.cpanel, .htpasswds, .trash, access-logs, etc, logs, mail etc) in the robots file? I have a feeling that this is not necessary as they are obviously protected with the password I use to access the control panel.

Secondly in the folder at path ftp.[mysite].myzen.co.uk.htpasswds/public_html  there are a number of folders some of which mirror the folder names of existing folders in ftp.[mysite].myzen.co.uk/public_html and some that are the names of old folders now renamed or deleted, that were directory protected at one stage. Can I delete the folders in the htpasswds folder that no longer exist in the public_html folder?  I am wonderin gif these old folders with password protection are affecting the functioning of the password protection on the folders that are still there.

Is there a guide that explains the Linux hosting folder structure?

Thanks for any help in advance.

blank_pageThe robots.txt should be placed inside the public_html as this is the only location which is publicly accessible by people browsing your site and as such to any spider agents indexing the site.  You do not need to exclude the .cpanel, .htpasswds, etc as they are not visible to the public.

If the folders in question are no longer being protected then the folders in ftp.[mysite].myzen.co.uk/.htpasswds can be deleted but they should have no impact on the functioning of any current password protection so ultimetly the choice is yours.

At current there is no guide to explain the folder structure but generally the only folder of concern is public_html (or www which is exactly the same) as the other folders are mostly maintained by the cPanel system.  However if you have any more questions please feel free to ask.

Daniel

Thanks for the quick reply.  Glad to say that I think I had nearly worked it out for myself but good to have it confirmed.

Another question that occurs to me is the functioning of some affiliate links.  I cannot get Adobe affiliate links to work.  They just show as dead.  IS this becuase those scripts need to be run from a specific folder?  Maybe cgi-bin...?

Regards

I'm not sure on the affiliate links as it's not something I have ever had any exposure to.  However if you PM some more details I can certainly take a look for you.