On our server I've been trying to run a Site Analysis report but it doesn't get beyond the site root due to a Robots.txt rule not allowing it, however there is no Robots.txt.
If I go to 'http://.../robots.txt' it does find a file, and it's as below,
User-agent: *
Disallow: /
But again, this file does not exist anywhere on the server, I've done a full search across all drives, all folders to be sure and there isn't a single Robots.txt on there, creating one with the correct content in it and placing that one in site root doesn't change anything and it continues to show the above.
I just can't figure out whats going on. Any ideas would be really appreciated.
Thanks
If I go to 'http://.../robots.txt' it does find a file, and it's as below,
User-agent: *
Disallow: /
But again, this file does not exist anywhere on the server, I've done a full search across all drives, all folders to be sure and there isn't a single Robots.txt on there, creating one with the correct content in it and placing that one in site root doesn't change anything and it continues to show the above.
I just can't figure out whats going on. Any ideas would be really appreciated.
Thanks