[NPMCLI-72] Research and document how users can test their modules Created: 17/Jan/17  Updated: 22/Feb/17  Resolved: 10/Feb/17

Status: Closed
Project: Magnolia CLI
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Neutral
Reporter: Robert Kowalski Assignee: Robert Kowalski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2017-02-10 at 10.48.09.png    
Template:
Acceptance criteria:
Empty
Task DoR:
Empty
Sprint: Basel 82
Story Points: 13

 Description   

Investigate and document how you can test light-modules, in a variety of contexts:

  • Ones managed with maven, as it has a nice functionality to run multiple Node and npm versions.
  • Ones not managed by maven - as light-modules are commonly developed by frontend developers with no maven installed.
  • For light-modules shared on github & npm.

A few questions:

  • What should be tested?
  • How will users create tests?
  • How difficult are the tests to maintain?
  • How can the CLI support this?

Rationale:
Tests ensure quality.
Tests demonstrate quality of projects on npm and github, and facilitate forking and innovation.



 Comments   
Comment by Robert Kowalski [ 03/Feb/17 ]

mgnl-custom is tested on travis now: https://travis-ci.org/robertkowalski/mgnl-custom

commit:

https://github.com/robertkowalski/mgnl-custom/commit/93032ac8b247f15f6163bcbd692637539b5c2f32

in this case i am using "jest" as test runner.

Comment by Robert Kowalski [ 03/Feb/17 ]

Testing Light Modules

Landscape

Testing is an interesting topic as it is so diverse and fragmented. Some developers, even with years of work experience, don't write tests at all for their software. How testing is done also depends what kind of Light Module we create.

Some test frameworks are better suited for different needs. As an example: Angular recommends Karma and Jasmine – the React project suggest Jest for testing. Often the choice of the test runner / test framework is also influenced by what the developer has used in the past. Other reasons can be limitations by other circumstances, e.g. legacy code.

Most projects on GitHub use Travis CI for continuous integration, but also there is a variety of possible freemium solutions, like Codeship. The services support test matrixes of multiple Node versions, so tests can be run on multiple Node versions.

While most projects use some sort of unit testing, integration tests (f.e. based on Selenium) are far more uncommon, at least for single libraries. Even for full applications they are not as common as unit tests, probably because they are hard to set up, debug and running slow.

Integration testing with Selenium and Magnolia

Components are set up using dialogs from the admin interface. I thought about running browser tests automated with Selenium, filling out the dialogs and then checking the result. I came to the conclusion that it would be of a very limited use. Running Selenium on the Admin UI would mostly test Magnolias functionality to show dialogs and render templates. It would also require a lot of ongoing maintenance, as soon as Magnolia changes the markup, the developer would have to fix the failing tests.

Another way would be to programatically create pages with the different Light Modules on it and then test the delivered functionality for the created (public) site. So far I did not get it running with a bootstrap.xml or the HTTP API.

What should be tested?

What should be tested varies from project to projects: the developer (some don't write tests at all), the complexity of the project and goals of the software.

Some components are more markup based, some are almost just made out of JavaScript.

It will be very different depending on the developer and the use case of the Light Module. Setting up the testing framework is a few lines of code, see https://github.com/robertkowalski/mgnl-custom/commit/93032ac8b247f15f6163bcbd692637539b5c2f32

How will users create tests?

In https://github.com/robertkowalski/mgnl-custom/commit/93032ac8b247f15f6163bcbd692637539b5c2f32 I installed a test runner and added tests for my Light Module. The way the Light Module is tested is similar to other frontend and node.js.

As the landscape for testing is so fragmented (different tooling and use cases) there is sadly no silver bullet.

The tooling part was easy for me. I was struggling to programatically create a test environment in Magnolia itself for integration testing. I tried the Magnolia REST API or other methods like decorators. I did not find a feasible solution.

Example from CouchDB to set up a test database and test document in a database (analog to pages and components):

# create a db
POST   http://localhost:5984/my_test_db {}

# create a document
POST   http://localhost:5984/my_test_db/my_test_doc {"food": ["pizza", "apple"]}

How can the CLI support this?

The npm ecosystem already provides everything (libraries, npm test command etc) needed. When people write tests for their frontend project, they will in 99.99% use npm to install the test runners.

If we can't improve the REST API / API docs directly we could somehow wrap the needed steps in the CLI as a last resort. In the perfect world the CLI wouldn't need to abstract and simplify it.

As part of my tests I spiked a command for the CLI ("mgnl get") which I demoed this Geeky Friday. When a package is on npm, its easy to get the package without a package JSON with the command. The default `node_modules` folder is not created in the process. Its compatible to npm and npm based builds, so packages can be used as pre-build standalone (mgnl get) and at the same time as part of a another main build (npm install).

Conclusion

Testing the JS in Light Modules is similar to other JS testing.

I was mainly struggling to programatically create a test environment in Magnolia.

The goal for me was to setup a Magnolia with my Modules on every CI run for Selenium based integration tests.

If we can simplify creating a page with a component using the REST API we can support testing with different Selenium based frameworks a lot better.

Comment by Robert Kowalski [ 06/Feb/17 ]

Possible test frameworks for integration testing:

http://www.protractortest.org/#/
https://theintern.github.io/intern/#writing-functional-test
http://nightwatchjs.org/

Super simple (incomplete) approach: we could also just request the http page and use https://github.com/cheeriojs/cheerio to traverse the DOM in a jquery like fashion. Problem is we are very limited (no DOM interactions like clicks or scrolls possible) and also CSS issues (box overlapping a link and not clickable) are not covered.

Comment by Robert Kowalski [ 08/Feb/17 ]

After discussion with Topher, mgnl-custom now supports integration testing with Travis CI.

Have to sync with Topher again today.

Comment by Robert Kowalski [ 08/Feb/17 ]

estimation: 1 more story point

Comment by Robert Kowalski [ 08/Feb/17 ]

Integration testing works great on travis, `cheerio` is our choice for template testing right now - further questions/tasks:

  • replace `mgnl jumpstart` based setup with a solution that enables to switch between magnolia versions. can we use a docker container or do we have an apt-mirror? travis supports apt-caching.

Tasks:

  • add cheerio based test
  • ask jan about the status docker-container
  • find out if there are apt-mirrors
  • adjust the example to them
  • replace grep with cheerio
  • document in wiki
Comment by Robert Kowalski [ 10/Feb/17 ]

light module unit & integration tests running in a test matrix with different magnolia versions on travis now:

https://github.com/robertkowalski/mgnl-custom/commit/9f7339b3888285b17b9c550604065db2523c25a7

Comment by Robert Kowalski [ 10/Feb/17 ]

results moved to https://wiki.magnolia-cms.com/display/DEVINT/Testing+Light+Modules+for+Frontend+Developers

scope grew over the week, re-estimate?

Generated at Mon Feb 12 04:46:15 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.