[MGNLEESOLR-160] Update crawler4j library Created: 30/Jul/21  Updated: 28/Jun/22  Resolved: 23/Mar/22

Status: Closed
Project: Solr Search Provider
Component/s: None
Affects Version/s: 5.6
Fix Version/s: 6.0

Type: Improvement Priority: Neutral
Reporter: Richard Gange Assignee: Milan Divilek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screenshot 2022-03-25 at 15.06.28.png    
Issue Links:
Relates
Template:
Acceptance criteria:
Empty
Task DoD:
[X]* Doc/release notes changes? Comment present?
[X]* Downstream builds green?
[X]* Solution information and context easily available?
[X]* Tests
[X]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Release notes required:
Yes
Documentation update required:
Yes
Date of First Response:
Epic Link: DevX Bucket
Sprint: DevX 6
Story Points: 2
Team: DeveloperX

 Description   

Currently we depend on version 4.1. Please update to 4.3 (or better) for better handling of robot.txt files. See https://github.com/yasserg/crawler4j/pull/78.



 Comments   
Comment by Javier Benito [ 25/Mar/22 ]

QA done:

 

 

 

Quick comment about how to setup Solr on Apple M1 chipset with OpenJDK. -Xss JVM option needs to have a higher value: https://stackoverflow.com/questions/70217229/solr-not-starting-on-macos-m1-using-azul-jvm 

Generated at Mon Feb 12 11:00:42 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.