loading
The biggest problem I faced was "Non spider friendly content" ...sigh..... was effecting all area's of the ranking procedure which spiders where using to catalog and rank your page/s.
Personally I use a homemade referral capture scheme, The following technique assumes you have the following files in web root - ror.xml , sitemap.html , sitemap.xml , urllist.txt
These files assure your site to be fully spider craw able, If you don’t have these files then make them or use a site like http://www.xml-sitemaps.com
Using the following system you will gather all referral traffic from other sites obviously, Point being spiders associate sites together for search queries based on external link matching, text on page (linked from your sitemaps) META tags specified on page + some other factors which combined completes your sites rank on the spider.
First thing to conceder is of course you wish to have your own keywords for your own site as specified as possible BUT you have a better of chance of trying to associate yourself with other sites which in turn when a user searches other site queries if your tied to the site in the search engine there is a good possibility of your own site appearing although nothing to do with the search.
I have been fully monitoring most search engines query patterns either from the source site admin panels or my own monitoring system and the method that works best is so stupidly simple.
Using the content URL’s captured from the referral system you need to create a simple page which spits out all the URL’s on page, Plain text only name it something like spider_content.PHP whatever then add the page into your sitemap content files (all 4) then wait (Mark Priority 8). I'm already appearing on search queries non related to my site as listed by Google admin panel itself from this method. My referral traffic is to say the least "higher" now.
Providing the content on spider_content is always self generating from the SQL tables you want then the search engine spider searches will always get the latest data including from content considered before "Non spider friendly"
I personally state on the top of my page :
MYSQL content for spiders vr 1.4 stonerscolony customs
The following text is MYSQL dump of content ON site but non spider friendly for regular search
// MYSQL TABLE ref
CREATE TABLE `ref` (
`group` varchar(200) collate latin1_general_ci NOT NULL,
`firstdate` int(15) NOT NULL,
`lastdate` int(15) NOT NULL,
`url` varchar(255) collate latin1_general_ci NOT NULL,
`hits` int(100) NOT NULL default '0',
`domain` varchar(200) collate latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
// The following code goes in your index page after DB Connection has been made
// Referral Capture ***
$ref = strtolower($_SERVER['HTTP_REFERER']);
preg_match("/^(http:\/\/)?([^\/]+)/i",
$ref, $matches);
$host = $matches[2];
// If you get sites duping from vdomain use
if($host=="www.exampleurl.com" OR $host=="exampleurl.com"){$host= "exampleurl";}
//redirect hated sites
if($host=="www.site.com" or $host=="vdomain.site.com"){
print("<meta http-equiv=refresh content='5;url=$host'>");
Print "<font size=3><center>No $host members thank you." ; die;}
// Stop self referrals
if($host=="www.YOURURL.com" or $host=="YOURURL.com"){ $ref="";}
if ($ref>"") { $now = time();
$r=0; $refq=mysql_query("SELECT * FROM `ref`");
$refc=mysql_numrows($refq);
while($r<$refc){ // check if domain has been captured before
$url=mysql_result($refq,$r,"domain");
if($url==$host){ $hityes = 1;}
$r++; }
if($hityes==1){
mysql_query("UPDATE `ref` SET hits = hits + 1, lastdate = '$now' WHERE `url` = '$ref'");
}else {
mysql_query("INSERT INTO `ref` VALUES ('$now', '$now', '$ref', '1', '$host')");
}
} // end referral capture
// search_engines.php example
//include session file here OR make DB Connection
print "<br><br><li>Sites linking to Your site URL<br><br>";
$q = mysql_query("SELECT * FROM `ref`");
$r= mysql_numrows($q);
$i=0;while($i<$r){
$domain = mysql_result($q,$i,"domain");
print " $domain - ";
$i++;}