[BUILD-576] Unavailable ND4J native libs fail dependency resolution on M1 Created: 08/Nov/21  Updated: 14/Nov/22  Resolved: 29/Apr/22

Status: Closed
Project: Build
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Mikaël Geljić Assignee: Andrei Ichimescu
Resolution: Won't Fix Votes: 1
Labels: authx
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive magnolia-customer-data-platform-demo.zip     Text File mvn_clean_install.log     Text File mvn_version.log    
Issue Links:
Relates
relates to MGNLPER-148 SIGILL - JVM core dump on M1 (apple s... Closed
relates to MGNLPER-154 Remove ranking from bundle Closed
relates to BUILD-749 Update DL4J version to the latest Closed
dependency
depends upon MGNLPER-156 DOC: Adjust list of modules page to s... Closed
relation
is related to MGNLPER-154 Remove ranking from bundle Closed
Template:
Acceptance criteria:
Empty
Date of First Response:
Epic Link: Remove ranking from bundle
Sprint: AuthX 8

 Description   

For release notes (see also MGNLPER-154 Remove ranking):

  • Remove ranking will be unbundled from DX Core
  • but still available to use, if there's a wish to do so
  • from version 6.2.x, and hence, it will come as unbundled in 6.3. branch (relevant for release notes) 

 

—on behalf of rfalvo, building the project archetype on a M1 Mac fails eagerly at dependency resolution time (even a dependency:tree fails), due to a non-existing nd4j native library for this platform.

this yields a javacpp.platform of “macosx-arm64” which causes maven + nd4j-native-platform to try to pull artifacts that don’t exist.

—via https://community.konduit.ai/t/support-for-apple-silicon-m1/1168

[ERROR] Failed to execute goal on project magnolia-customer-data-platform-demo-webapp: Could not resolve dependencies for project info.magnolia.demo:magnolia-customer-data-platform-demo-webapp:war:1.0-SNAPSHOT: The following artifacts could not be resolved: org.nd4j:nd4j-native:jar:macosx-arm64:1.0.0-beta7, org.bytedeco:javacpp:jar:macosx-arm64:1.5.3, org.bytedeco:openblas:jar:macosx-arm64:0.3.9-1-1.5.3: org.nd4j:nd4j-native:jar:macosx-arm64:1.0.0-beta7 was not found in https://nexus.magnolia-cms.com/content/groups/staff during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of magnolia.nexus has elapsed or updates are forced -> [Help 1]


 Comments   
Comment by Raphael Falvo [ 08/Nov/21 ]

A workaround consists of excluding the ranker module from the Maven dependencies (thanks mgeljic for the tip) until those missing libraries exist for M1

In pom.xml of the web application, add the exclusions as below:

<dependencies>
  <dependency>
    <groupId>info.magnolia.dx</groupId>
    <artifactId>magnolia-dx-core-demo-webapp</artifactId>
    <type>war</type>
    <exclusions>
      <exclusion>
        <groupId>info.magnolia.periscope</groupId>
        <artifactId>magnolia-periscope-result-ranker</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
  <dependency>
    <groupId>info.magnolia.dx</groupId>
    <artifactId>magnolia-dx-core-demo-webapp</artifactId>
    <type>pom</type>
    <exclusions>
      <exclusion>
        <groupId>info.magnolia.periscope</groupId>
        <artifactId>magnolia-periscope-result-ranker</artifactId>
      </exclusion>
    </exclusions>
  </dependency>

 

Comment by Mikaël Geljić [ 29/Mar/22 ]

Reopening because this will only be solved when we remove the traces of DL4J/ND4J in the 3rd-party BOM (what causes the platform-specific maven resolution to fail).

Should be all lines 2054 to 2160 (based on master/6.3-SNAPSHOT), https://git.magnolia-cms.com/projects/BUILD/repos/boms/browse/pom.xml#2054-2160, except commons-net, which got inserted in the middle.

Comment by Andrei Ichimescu [ 29/Apr/22 ]

According to the library documentation (1.0.0-M2) the supported architectures do not include apple silicon:

All projects in the DL4J ecosystem support Windows, Linux and macOS. Hardware support includes CUDA GPUs (10.0, 10.1, 10.2 except OSX), x86 CPU (x86_64, avx2, avx512), ARM CPU (arm, arm64, armhf) and PowerPC (ppc64le).

https://github.com/eclipse/deeplearning4j/pull/9631#community-support

there is a feature request that has never been addressed:
https://community.konduit.ai/t/support-for-apple-silicon-m1/1168

 people managed to get this running by changing the jdk according to the thread.

 

Since the periscope module has been donated to the community, I would mark this ticket as won't fix

Generated at Sun Feb 11 23:43:09 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.