[MGNLSYNC-40] Add support for invoking synchronization asynchronously Created: 06/Jan/20  Updated: 29/Mar/22  Resolved: 27/Apr/20

Status: Closed
Project: Magnolia Synchronization Module
Component/s: None
Affects Version/s: None
Fix Version/s: 1.9

Type: Improvement Priority: Neutral
Reporter: Jan Haderka Assignee: Andrew Warinner
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 4h 35m
Time Spent: 5h 40m
Original Estimate: Not Specified

Issue Links:
causality
is causing MGNLSYNC-43 Finalize support for invoking synchro... Closed
dependency
is depended upon by MGNLSYNC-41 Add support for parallelization of sy... Open
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:
Team: Nucleus

 Description   

Currently when invoking synchronization from headless environment, via REST Command API, one has to keep request open for command to run. For longer execution REST call would time out and cause command execution to be interrupted. Since it is also not possible to monitor the command execution to know how much more content has to be still synchronized (due to nature of synchronization), it makes it also very difficult to configure timeout of the REST call correctly.

 

Proposal:

  • add support for async invocation (e.g. via using scheduled task with near-immediate execution) to start command itself asynchronously and only return some kind of job ID.
  • add REST API/Servlet to check the status of the job by specifying previously provided job ID. It would be enough to provide JSON statuses such as NOT_STARTED, RUNNING, DONE, FAILURE, with more detailed error message for the last one so that one can check the job status periodically.


 Comments   
Comment by Mikaël Geljić [ 08/Jan/20 ]

proposing a micro, domain-specific REST API, here's some rough prototyping:

  • PUT /.rest/synchronizations (or /sync/trigger) returns 201 with a synchonizationId (impl-wise can be composed of scheduler job id)
  • GET /.rest/synchronizations/{synchronizationId} with proper sub-resource (by default we may want /status, might as well have another method-endpoint for /abort)

as discussed awarinner, should give already more control for ops & tooling.

Comment by Jan Haderka [ 08/Jan/20 ]

In order to invoke job asynchronously, we propose to start the actual command via scheduled task. This will simplify achieving requirements specified in MGNLSYNC-41

Comment by Jan Haderka [ 08/Jan/20 ]

Alternatively:

  • in the above REST call (/sync/trigger) make sure that it can be called with all the details used to create receiver info (url, workspace, topmost node for sync, time from which to sync) and as a first step create such receiver configuration under synchronization.
  • in the second step you would then call JobBuilder to create and invoke new scheduled job:
final JobDetail jd = JobBuilder.newJob()
        .withIdentity(jobName, SchedulerConsts.SCHEDULER_GROUP_NAME)
        .ofType(info.magnolia.module.scheduler.CommandJob.class)
        .build();
jd.getJobDataMap().put(SchedulerConsts.CONFIG_JOB_COMMAND, commandName);
jd.getJobDataMap().put(SchedulerConsts.CONFIG_JOB_COMMAND_CATALOG, catalogName);
jd.getJobDataMap().put(SchedulerConsts.CONFIG_JOB_PARAMS, params);Scheduler scheduler = schedulerModuleProvider.get().getScheduler();
TriggerListener triggerListener = getListener(jobName, item);
scheduler.getListenerManager().addTriggerListener(triggerListener);
try {
    scheduler.scheduleJob(jd, trigger);
} catch (SchedulerException e) {
    throw new ParallelExecutionException(e);
}
 

Where commandName and catalogName would point to custom SynchronizationCommand and you would provide it with name of receiver it should use over params

 

Your custom job just need to create and configure single receiver based on the name it gets over params instead of relying on all receivers to be set via CommandsManager

 

That said and done should actually allow convenient execution of synchronization w/o any need for MGNLSYNC-41 since each all will be limited to single subscriber and when other part of this ticket is implemented, user would know whether sync job for given url (one provided as param to /trigger call) is still running or done already.

Comment by Jan Haderka [ 08/Jan/20 ]

And to get the status of the job for the other part of this ticket, it should be as simple as:


schedulerModuleProvider.get().getScheduler().getTriggerState(trigger)
 

 

Trigger the call takes as param is object created when scheduling the job and should be possible to recreate it with it's ID. Returned trigger state is then enumeration that holds the current state of job with given trigger.

Comment by Jan Haderka [ 08/Jan/20 ]

One extra thing to note and check is that currently{{SynchrionizationCommand}} will synchronize only one workspace at the time. It might be advisable to allow a configration/call over REST API that would trigger sync of ALL workspaces.

Comment by Mikaël Geljić [ 08/Jan/20 ]

API-wise might turn into /trigger/{workspace}{path:(/.+)? + eventually /trigger/all re: synchronizing all-at-once

Comment by Mikaël Geljić [ 03/Feb/20 ]

Successfully merged basic functionality and released a 1.9-beta1; Here are additional review items for 2nd-stage review:

  • REST API review - not a problem to change provided we produce a change log / release notes
  • Rename 'repository' params back to 'workspace'
  • Date resolution for the endpoint (help define expected ISO date string format?)
  • Unbox booleans where applicable
  • VH for module config and command bootstraps? non-applicable for new module
  • Move manager to synchronization-core
  • Builder for SynchronizationRequest + pass as param to #synchronize(SyncRequest), #canSynchronize...
  • Remove XML output from endpoint (non-goal)
  • Consider ITs for endpoint?
  • Test method signatures: throw general Exception & no test prefix (BDD-style naming)
  • Failing existing synchronization command tests – failing on JDK12, passing on JDK8
  • Validate module config vs. manager, drop module lifecycle?
  • Reconsider deduplicating `SynchronizationCommand` (preferably evolving or partially deprecating the old one)
  • Global policy?
Comment by Simon Lutz [ 27/May/20 ]

As this ticket was recently closed, I changed the fix version from 1.9 to 1.9-beta2 and filed the finalization as MGNLSYNC-43.

Generated at Mon Feb 12 07:39:16 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.