I have the following code in robots.txt to allow crawling from everyone for now
User-agent: *
Disallow:
Before I changed this the layout of the file was this below. I've been looking for details about unique and I can't find it. Anyone see this before and what is "unique" doing exactly?
User-agent: *
Disallow: /unique/
It's not a keyword, it's a directory on your server that shouldn't be visited by a web crawler.