Class Web.Crawler.Policy

Description

The crawler policy object.

Variable bandwidth_throttling_floating_window_width

int Web.Crawler.Policy.bandwidth_throttling_floating_window_width

Description

Bandwidth throttling floating window width. Defaults to 30.

Variable max_bits_per_second_per_host

int Web.Crawler.Policy.max_bits_per_second_per_host

Description

Maximum number of bits per second, per host. Defaults to off (0).

Variable max_bits_per_second_total

int Web.Crawler.Policy.max_bits_per_second_total

Description

Maximum number of bits per second. Defaults to off (0).

Variable max_concurrent_fetchers

int Web.Crawler.Policy.max_concurrent_fetchers

Description

Maximum number of fetchers. Defaults to 100.

Variable max_concurrent_fetchers_per_host

int Web.Crawler.Policy.max_concurrent_fetchers_per_host

Description

Maximum concurrent fetchers per host. Defaults to 1.

Variable min_delay_per_host

int Web.Crawler.Policy.min_delay_per_host

Description

Minimum delay per host. Defaults to 0.