IgnoreRobotProtocolErrors

A Boolean value that specifies whether to ignore robot protocol errors.

When you set FollowRobotProtocol to TRUE, Web Connector requests a file named robots.txt from the root directory of a web site. The web server should return a correctly formatted robots.txt or should respond with an HTTP 404 (not found) response code. If the server gives any other response, for example returning HTTP 200 (success) but providing a malformed file, Web Connector attempts to retrieve the robots.txt file again before each page request.

To ignore invalid robots protocol files and synchronize the web site as if no robots.txt had been provided, set this parameter to TRUE.

Type: Boolean
Default: False
Required: No
Configuration Section: TaskName or FetchTasks
Example: IgnoreRobotProtocolErrors=TRUE
See Also: FollowRobotProtocol