Hello Search-This readers, I got a question for ya. I don’t know if you’ll know the answer, but I don’t know what’s up either…
I have 3 directories outside of my wp-content directory and it appears that Google can’t index them?
The directories include:
http://www.search-this.com/search-engine-decoder/
http://www.search-this.com/pagerank-decoder/
http://www.search-this.com/tools/
This is my robots.txt file:
User-agent: *
Disallow: /cgi-bin/
Disallow: /logs/
Disallow: /wp-admin/
Disallow: /wp-includes/
What’s up? How come Google can’t index what’s outside of wp-content ?
thanks,
Mark
Here’s my .htaccess file
- RewriteEngine On
- # trim index.html
- RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*)index\.html\ HTTP/
- RewriteRule index\.html$ http://www.search-this.com/%1 [R=301,L]
- #
- # trim index.php
- RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*)index\.php\ HTTP/
- RewriteRule index\.html$ http://www.search-this.com/%1 [R=301,L]
- #
- # redirect from non www to www
- RewriteCond %{HTTP_HOST} ^search-this\.com
- RewriteRule ^/?(.*)$ http://www.search-this.com/$1 [R=301,L]
- # START Prevent HotLinking
- RewriteCond %{HTTP_REFERER} !^$
- RewriteCond %{HTTP_REFERER} !^http://(www\.)?search-this.com/.*$ [NC]
- RewriteRule \.(gif|jpg|js|css|swf)$ - [F]
- # END Prevent HotLinking
- # DIRECTORIES
- Redirect 301 /search_engine_decoder.asp http://www.search-this.com/search-engine-decoder/
- Redirect 301 /search-engine-decoder.swf http://www.search-this.com/search-engine-decoder/
- Redirect 301 /search_engine_optimization/ http://www.search-this.com/search-engine-optimization/
- Redirect 301 /search_engine_optimization http://www.search-this.com/search-engine-optimization/
- Redirect 301 /search_engine_submission/ http://www.search-this.com/search-engine-optimization/
- Redirect 301 /website_design/ http://www.search-this.com/website-design/
- Redirect 301 /website_design http://www.search-this.com/website-design/
- Redirect 301 /website_promotion/ http://www.search-this.com/website-promotion/
- Redirect 301 /website_promotion http://www.search-this.com/website-promotion/
- Redirect 301 /search_engine_articles/ http://www.search-this.com/
- Redirect 301 /search_engine_articles http://www.search-this.com/
- Redirect 301 /forums/ http://www.search-this.com/
- Redirect 301 /forums http://www.search-this.com/
- Redirect 301 /pagerank_decoder/ http://www.search-this.com/pagerank-decoder/
- Redirect 301 /pagerank_decoder http://www.search-this.com/pagerank-decoder/
- # PROTECT
- <Files .htaccess>
- Order allow,deny
- Deny from all
- </Files>
- # BEGIN WordPress
- <IfModule mod_rewrite.c>
- RewriteBase /
- RewriteCond %{REQUEST_FILENAME} !-f
- RewriteCond %{REQUEST_FILENAME} !-d
- RewriteRule . /index.php [L]
- </IfModule>
- # END WordPress
January 26th, 2007 at 12:25 pm
Do you have them set up as “WordPress Pages” or are they just regular Web pages?
January 26th, 2007 at 1:18 pm
Good question Dan, they are ‘regular’ web pages. But, I do put this at the top of them:
< ?php define('WP_USE_THEMES', false); require($_SERVER['DOCUMENT_ROOT']."/wp-blog-header.php"); ?>
this then allows me to use things like: < ?php bloginfo('template_directory'); ?>
But, I don’t know maybe this is the problem? I can play with that….
January 27th, 2007 at 11:27 pm
I have no clue why Google can’t index those 3 pages, but it’s now chapping my hide.
I’ll got $10 bucks to anyone who can find the problem…
January 28th, 2007 at 2:34 pm
Maybe this is a shot in the dark, but would your .htaccess file that does the Mod-Rewrite hack for the friendly URLs for posts have any bearing on it?
January 28th, 2007 at 4:19 pm
Hey cp, I don’t think it’s the .htaccess file. I don’t see anything in there that would do that but I have posted it above – have a look.
January 28th, 2007 at 8:57 pm
Well, I question whether the wordpress .htaccess info that checks the directories after the domain name and directs them to index.php would cause the problem.
Though it looks like the !-d would ensure its not an existing directory already. So thus, nothing seems out of the ordinary to me.