January 26th, 2007 - by Golgotha

Hello Search-This readers, I got a question for ya. I don’t know if you’ll know the answer, but I don’t know what’s up either…

I have 3 directories outside of my wp-content directory and it appears that Google can’t index them?

The directories include:
http://www.search-this.com/search-engine-decoder/
http://www.search-this.com/pagerank-decoder/
http://www.search-this.com/tools/

This is my robots.txt file:
User-agent: *
Disallow: /cgi-bin/
Disallow: /logs/
Disallow: /wp-admin/
Disallow: /wp-includes/

What’s up? How come Google can’t index what’s outside of wp-content ?

thanks,
Mark

Here’s my .htaccess file

  1. RewriteEngine On
  2.  
  3. # trim index.html
  4. RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*)index\.html\ HTTP/
  5. RewriteRule index\.html$ http://www.search-this.com/%1 [R=301,L]
  6. #
  7. # trim index.php
  8. RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*)index\.php\ HTTP/
  9. RewriteRule index\.html$ http://www.search-this.com/%1 [R=301,L]
  10. #
  11. # redirect from non www to www
  12. RewriteCond %{HTTP_HOST} ^search-this\.com
  13. RewriteRule ^/?(.*)$ http://www.search-this.com/$1 [R=301,L]
  14.  
  15. # START Prevent HotLinking
  16. RewriteCond %{HTTP_REFERER} !^$
  17. RewriteCond %{HTTP_REFERER} !^http://(www\.)?search-this.com/.*$ [NC]
  18. RewriteRule \.(gif|jpg|js|css|swf)$ - [F]
  19. # END Prevent HotLinking
  20.  
  21. # DIRECTORIES
  22. Redirect 301 /search_engine_decoder.asp http://www.search-this.com/search-engine-decoder/
  23. Redirect 301 /search-engine-decoder.swf http://www.search-this.com/search-engine-decoder/
  24. Redirect 301 /search_engine_optimization/ http://www.search-this.com/search-engine-optimization/
  25. Redirect 301 /search_engine_optimization http://www.search-this.com/search-engine-optimization/
  26. Redirect 301 /search_engine_submission/ http://www.search-this.com/search-engine-optimization/
  27. Redirect 301 /website_design/ http://www.search-this.com/website-design/
  28. Redirect 301 /website_design http://www.search-this.com/website-design/
  29. Redirect 301 /website_promotion/ http://www.search-this.com/website-promotion/
  30. Redirect 301 /website_promotion http://www.search-this.com/website-promotion/
  31. Redirect 301 /search_engine_articles/ http://www.search-this.com/
  32. Redirect 301 /search_engine_articles http://www.search-this.com/
  33. Redirect 301 /forums/ http://www.search-this.com/
  34. Redirect 301 /forums http://www.search-this.com/
  35. Redirect 301 /pagerank_decoder/ http://www.search-this.com/pagerank-decoder/
  36. Redirect 301 /pagerank_decoder http://www.search-this.com/pagerank-decoder/  
  37.  
  38. # PROTECT
  39. <Files .htaccess>
  40. Order allow,deny
  41. Deny from all
  42. </Files>
  43.  
  44. # BEGIN WordPress
  45. <IfModule mod_rewrite.c>
  46.  
  47. RewriteBase /
  48. RewriteCond %{REQUEST_FILENAME} !-f
  49. RewriteCond %{REQUEST_FILENAME} !-d
  50. RewriteRule . /index.php [L]
  51. </IfModule>
  52. # END WordPress

6 Responses to “Help – Google Can’t Index Outside of wp-content directory?”

1 Dan Schulz

Do you have them set up as “WordPress Pages” or are they just regular Web pages?

2 Golgotha

Good question Dan, they are ‘regular’ web pages. But, I do put this at the top of them:

< ?php
define('WP_USE_THEMES', false);
require($_SERVER['DOCUMENT_ROOT']."/wp-blog-header.php");
?>

this then allows me to use things like: < ?php bloginfo('template_directory'); ?>

But, I don’t know maybe this is the problem? I can play with that….

3 Golgotha

I have no clue why Google can’t index those 3 pages, but it’s now chapping my hide.

I’ll got $10 bucks to anyone who can find the problem…

4 cpradio

Maybe this is a shot in the dark, but would your .htaccess file that does the Mod-Rewrite hack for the friendly URLs for posts have any bearing on it?

5 Golgotha

Hey cp, I don’t think it’s the .htaccess file. I don’t see anything in there that would do that but I have posted it above – have a look.

6 cpradio

Well, I question whether the wordpress .htaccess info that checks the directories after the domain name and directs them to index.php would cause the problem.

Though it looks like the !-d would ensure its not an existing directory already. So thus, nothing seems out of the ordinary to me.

mulberry sale spyder womens jacket cheap new balance 574 mulberry outlet cheap new balance 574 arcteryx outlet mulberry sale spyder womens jacket mulberry sale spyder womens jacket mulberry outlet mulberry outlet new balance 574

Popular Articles

Top 10 Commentators


Subscribe to this feed! Subscribe by Email!

Random Bits Podcast

You need to download the Flash player from Adobe

Blogs Worth Reading


    Warning: fopen(./wp-content/themes/big-blue/my_blogroll.php): failed to open stream: Permission denied in /home/mangeletti/search-this.com/wp-content/themes/big-blue/get_blogroll.php on line 5

    Warning: fread() expects parameter 1 to be resource, boolean given in /home/mangeletti/search-this.com/wp-content/themes/big-blue/get_blogroll.php on line 6

    Warning: fclose() expects parameter 1 to be resource, boolean given in /home/mangeletti/search-this.com/wp-content/themes/big-blue/get_blogroll.php on line 7
  • Smashing Magazine