-
Improvement
-
Resolution: Fixed
-
Neutral
-
5.6
-
None
-
-
Empty show more show less
-
Yes
-
Yes
-
DevX 6
-
2
Currently we depend on version 4.1. Please update to 4.3 (or better) for better handling of robot.txt files. See https://github.com/yasserg/crawler4j/pull/78.
Acceptance criteria