FollowRobotProtocol

A Boolean value that specifies whether the connector respects instructions provided by Web sites to Web crawlers. By default, the connector respects the following instructions:

To ignore these instructions, set FollowRobotProtocol=False.

Type: Boolean
Default: True
Required: No
Configuration Section: TaskName or FetchTasks or Default
Example: FollowRobotProtocol=False
See Also:  

_HP_HTML5_bannerTitle.htm