This is so far the most time consuming part of the entire idea. Because I am only indexing sites on the topic of speedway I cannot just allow the spider to follow links. As I would end up with lots of random websites.
Therefore I most first find the sites then tell the system to index them. This can take quite a while and already sites are running out of useable links.
I am having to use much harder to navigate site directorys (like a yellow pages)
Keep you updated… 