[Top]
Web
Web.Crawler
Web.Crawler.Policy
|
Class Web.Crawler.Policy
- Description
The crawler policy object.
- Variable
max_concurrent_fetchers
int Web.Crawler.Policy.max_concurrent_fetchers
- Description
Maximum number of fetchers. Defaults to 100.
- Variable
max_bits_per_second_total
int Web.Crawler.Policy.max_bits_per_second_total
- Description
Maximum number of bits per second. Defaults to off (0).
- Variable
max_bits_per_second_per_host
int Web.Crawler.Policy.max_bits_per_second_per_host
- Description
Maximum number of bits per second, per host. Defaults to off (0).
- Variable
bandwidth_throttling_floating_window_width
int Web.Crawler.Policy.bandwidth_throttling_floating_window_width
- Description
Bandwidth throttling floating window width. Defaults to 30.
- Variable
max_concurrent_fetchers_per_host
int Web.Crawler.Policy.max_concurrent_fetchers_per_host
- Description
Maximum concurrent fetchers per host. Defaults to 1.
- Variable
min_delay_per_host
int Web.Crawler.Policy.min_delay_per_host
- Description
Minimum delay per host. Defaults to 0.
|