Limit the Download Rate
To reduce the load on the Web site you are indexing, you can limit the maximum rate at which the connector downloads content, using the configuration parameter MaxKBytesPerSec
. This parameter does not limit the download rate for any individual page, it limits the maximum download rate by pausing between pages.
A Web server might be configured to deny requests that arrive from the same host with little or no time between them. The connector requests information from a Web site much faster than a human visitor, so you might find that the connector's requests are denied. To prevent a server from denying requests from the connector, you can specify a time interval that a synchronize thread must wait between requests. To do this, set the configuration parameter PageDelay
. If you set a delay of one second, each synchronize thread must wait at least one second between requests. By default, the connector uses five synchronize threads so might request up to five pages per second.
You can also limit the maximum number of pages that are ingested in a single synchronize cycle, by setting the configuration parameter MaxPages
. If the limit is reached, the remaining pages are ingested in future synchronize cycles.
TIP: For more information about these configuration parameters, refer to the Web Connector Reference.