Heritrix

This is a 'micro' release with bugfixes and small requested improvements. The next major release will be 2.2 in 2009, which is planned to include updates to the Heritrix 2 configuration system and checkpointing functionality, and tools easing transition from 1.14.x to Heritrix 2.2.

This release offers improved remote control and
monitoring via JMX, a crawl-checkpointing
facility, experimental support for bloom filter
already-included testing, partitioning a crawl
across multiple independent crawlers, and
per-host/domain/queue-grouping collection quotas.
Performance and stability in large crawls was
improved. 39 requested enhancements were included
and 96 reported bugs were fixed. You will need to
tweak your old order files again to make them work
with the new release.

This release features a much improved memory usage, a new experimental scoping/filter model, and a new revisiting frontier. Over 90 bugs were fixed.

This release adds IP-based politeness, configurable URI-
canonicalization, and mid-fetch abort. There were also lots of
bugfixes.

Crawl.log and ARC metadata lines could previously have whitespace in URIs and MIME etype fields.