A Boolean value that specifies whether the connector respects instructions provided by Web sites to Web crawlers. By default, the connector respects the following instructions:
robots.txt
, if present in the root directory of a Web site.To ignore these instructions, set FollowRobotProtocol=False
.
Type: | Boolean |
Default: | True |
Required: | No |
Configuration Section: | TaskName or FetchTasks or Default |
Example: | FollowRobotProtocol=False
|
See Also: |
|