Adobe AEM - From Eventing to Job Processing

FROM EVENT ING TO JOB PROCESS ING

Carsten Ziegeler | Adobe Research Switzerland

1

2

•  RnD Team at Adobe Research Switzerland •  Co-founder Adobe Granite

•  OSGi Core Platform and Enterprise Expert Groups •  Member of the ASF

•  Current PMC Chair of Apache Sling •  Apache Sling, Felix, ACE

•  Conference Speaker •  Technical Reviewer •  Article/Book Author

[email protected] @cziegeler

ABOUT

3

OVERVIEW Where to fit in the stack

Discovery

Offloading Workflow Distribution

DAM Ingestion

Sling

Granite

AEM

Job Distribution

Felix OSGi Event Admin

Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

4


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

5

OSGI EVENT ADMIN Publish / Subscribe Model

OSGi Event Admin Component

A

publish deliver

Component X

Component Y

6

•  OSGi event is a data object with •  Topic (hierarchical namespace) •  Properties (key-value-pairs)

•  Page Event •  Topic: com/day/cq/wcm/core/page •  Properties: path, change type (add/remove/edit) etc.


7

•  Publisher creates event object •  Sends event through EventAdmin service •  Either sync or async delivery

•  Subscriber is an OSGi service (EventHandler) •  Service registration properties •  Interested topic(s)

•  com/day/cq/wcm/core/* •  Additional filters (optional)

•  (type="add“)


8

•  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery


9

•  Immediate delivery to available subscribers •  No guarantee of delivery •  No distributed delivery


Discovery Sling Job Distribution

10


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

11


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution New in 5.6.1

12

INSTALLAT ION SCENARIOS

Clustered CRX

Single Instance

CRX

Instance 1

Instance 2

Instance 3

13

Instance: Unique Id (Sling ID)

TOPOLOGIES I Apache Sling Discovery

Clustered CRX CRX

ID : A ID : X ID : 42 ID : 1

Single Instance

Instance 1

Instance 2

Instance 3

14

•  Instance: Unique Id (Sling ID) •  Cluster: Unique Id and leader


Cluster 99

Cluster 35

Clustered CRX CRX

ID : A ID : X ID : 42 ID : 1

Single Instance

Instance 1

Instance 2

Instance 3

Leader Leader

15

•  Topology: Set of clusters


Cluster 99

Cluster 35

Clustered CRX CRX

ID : A ID : X ID : 42 ID : 1

Single Instance

Instance 1

Instance 2

Instance 3

Leader Leader

Topology Topology


16



•  Topology: Set of clusters

Cluster 99

Clustered CRX CRX

ID : A ID : X ID : 42 ID : 1

Single Instance

Instance 1

Instance 2

Instance 3

Leader

Topology

Cluster 35

Leader

Cluster 99

17

•  Instance •  Sling ID •  Optional: Name and description •  Belongs to a cluster •  Might be the cluster leader •  Additional properties which are distributed

•  Extensible through own services (PropertyProvider) •  E.g. data center, region or enabled job topics

•  Cluster •  Elects (stable) leader •  Stable instance ordering

•  TopologyEventListener: receives events on topology changes

TOPOLOGIES I I Apache Sling Discovery

ID : 42

Instance 3

Topology

Leader

18


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

19

•  Job : Guaranteed processing, exactly once •  Exactly one job consumer

•  Started by client code, e.g. for replication, workflow... •  Job topic •  Payload is a serializable map

JOB HANDLING I Apache Sling Job Handling

20

•  Sling Job Manager handles and distributes jobs •  Delivers job to a job consumer… •  …and waits for response •  Retry and failover

•  Notification listeners (fail, retry, success)

JOB HANDLING I Apache Sling Job Handling

21

S TART ING / PROCESS ING A JOB I Apache Sling Job Handling

public interface JobConsumer { String PROPERTY_TOPICS = "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC } JobResult process(Job job);}

public interface JobManager { Job addJob(String topic, String optionalName, Map<String, Object> properties); …}

Starting a job

Processing a job

Note: Starting/processing of jobs through Event Admin is deprecated but still supported

New in 5.6.1

22

@Component@Service(value={JobConsumer.class})@Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup")public class BackupJobConsumer implements JobConsumer { @Override public JobResult process(final Job job) { // do backup return JobResult.OK; }}

S TART ING / PROCESS ING A JOB I I Apache Sling Job Handling

New in 5.6.1

23

•  New jobs are immediately persisted (resource tree / repository) •  Jobs are “pushed” to the processing instance •  Processing instances use different queues

•  Associated with job topic(s) •  Main queue •  0..n custom queues

•  For example: replication agent queue or workflow queue

JOB HANDLING I I Apache Sling Job Handling

24

•  Queue is configurable •  Queue is started on demand in own thread

•  And stopped if unused for some time •  Types

•  Ordered queue (eg replication) •  Parallel queues: Plain and Topic Round Robin (eg workflow)

•  Limit for parallel threads per queue •  Number of retries (-1 = endless) •  Retry delay •  Thread priority

JOB QUEUE Apache Sling Job Handling

25

•  Job Manager Configuration = Main Queue Configuration •  Maximum parallel jobs (15) •  Retries (10) •  Retry Delay

•  Eventing Thread Pool Configuration •  Used by all queues •  Pool size (35) = Maximum parallel jobs for a single instance

ADDIT IONAL CONFIGURAT IONS Apache Sling Job Handling

26

MONITORING – WEB CONSOLE Apache Sling Job Handling

27

MONITORING – WEB CONSOLE Apache Sling Job Handling

28

•  Each instance determines enabled job topics •  Derived from Job Consumers (new API required) •  Can be whitelisted/blacklisted (in Job Consumer Manager) •  Announced through Topology

•  Job Distribution depends on enabled job topics and queue type •  Potential set of instances derived from topology (enabled job topics) •  Ordered : processing on leader only, one job after the other •  Parallel: Round robin distribution on all potential instances

•  Local cluster instances have preference

•  Failover •  Instance crash: leader redistributes jobs to available instances

•  Leader change taken into account

•  On enabled job topics changes: potential redistribution

JOB DISTR IBUT ION Apache Sling Job Handling

29

•  Scalability in AEM: •  DAM Ingestion •  Non-Clustered installation requirement

•  The term Offloading: •  In AEM used for all things job distribution and topology in clustered and non-

clustered installations, e.g. ‘Offloading Browser’ •  More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution

for handling non-clustered installations

WHY OFFLOADING

30


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

31


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

32

SL ING JOB DISTR IBUT ION

Topology

Instance Sling ID: 1

Job Manager


Job Manager


Job Manager


Job Manager

Job Consumer Topic: A

Job Consumer Topic: B

Job Consumer Topic: C

33



Job Manager


Job Manager


Job Manager


Job Manager A



A:2

Job Consumer Topic: C Job

Topology

34



Job Manager


Job Manager


Job Manager


Job Manager B



B:3


Topology

35



Job Manager


Job Manager


Job Manager


Job Manager C



C:4


?

Topology

36



Job Manager


Job Manager


Job Manager


Job Manager C



C:4


Offloading Framework

Topology

37

•  Detects offloading jobs •  Transport of job and job

payload between origin and target instance

•  Uses replication for the transport

•  No distribution of jobs •  No execution of jobs

OFFLOADING FRAMEWORK Overview


Job Manager

Offloading Job Manager

C:4


Job Manager

Job Consumer Topic: C

Offloading Job Manager

C:4

C

Job

Replication

1

2

3 5

4

6 7

Topology

38

•  Replication agents are created automatically •  Uses naming convention •  Needs manual adjustments (replication user)

OFFLOADING FRAMEWORK Replication

Worker Sling ID: 4

Author Sling ID: 1

offloading_4 (outgoing agent)

offloading_reverse_4 (reverse agent)

offloading_outbox (outbox agent)

39

•  Additional properties required •  Offloading job input

•  Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload) •  Offloading job output

•  Property: OffloadingJobProperties. OUTPUT_PAYLOAD (offloading.output.payload)

•  Takes comma separated list of paths •  Used to build the replication package •  Job sender needs to set these properties for offloading to work!

OFFLOADING FRAMEWORK Job Payload

40

•  Configures Job Consumers •  Configures the topic white/black listing properties of each instance •  What jobs to execute on what instance

•  Configures the distribution •  Configuration applies for both, clustered and non-clustered installations

OFFLOADING BROWSER UI

41


42


43


44


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

45

•  New JobConsumer •  Class: WorkflowOffloadingJobConsumer •  Topic: com/adobe/granite/workflow/offloading •  Can launch new workflows •  Expects the workflow model on the job payload •  Expects the workflow payload on the job payload

•  For use with clustered and non-clustered installations

WORKFLOW DISTR IBUT ION

46

@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }

WORKFLOW DISTR IBUT ION Job Consumer (Simplified)

47



•  Define the job topic

48



•  Create service component •  Must register with topic •  Implement new JobConsumer

interface

49



•  Access job properties (payload) •  Read workflow model and

payload from job properties

50



•  Workflow specific •  Use workflow API to start

workflow for the given model and payload

51



•  Use JobResult enumeration to report back the job status

52


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

53

•  Default ingestion workflow: “DAM Update Asset” •  Load is put on the instance where the workflow is started, usually the author

•  New ingestion workflow: “DAM Update Asset Offloading” •  Needs to be manually enabled by changing the workflow launcher •  New workflow model with a single step: AssetOffloadingProcess •  Uses WorkflowExternalProcess API •  Creates a new job on topic: com/adobe/granite/workflow/offloading •  Allows distributing the default ingestion workflow •  Load is put on the instance where the job is distributed to

•  Can be used to distribute in clustered and non-clustered installations

DAM INGEST ION

54

@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }

DAM INGEST ION Create Job (from workflow step)

55



•  Create service component •  Implement

WorkflowExternalProcess interface

•  Reference JobManager service

56



•  DAM and Workflow specific •  Resolve to Asset •  Read model from meta data •  Read workflow payload from

Asset path

57

public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }


•  ValueMap for job properties •  Put model and payload on job

properties •  Used by the JobConsumer

58



•  Build offloading payload properties •  Comma separated list of paths •  Put them on the job payload as

well •  Only used for non-clustered

distribution

59



•  Create job using JobManager service

•  Use topic from job consumer •  Put job payload properties •  Return the jobId as the workflow

process id (workflow specific)

60

@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ …. } public boolean hasFinished(Serializable externalProcessId, ..){ // returns null, if job is finished Job offloadingJob = jobManager.getJobById((String) externalProcessId); return offloadingJob == null; } }


•  Workflow API specific callback •  Process id = jobId, from

execute() •  Query job by jobId •  Workflow step finished when job

is finished

61

•  Choose a job topic •  Create JobConsumer component and register with topic chosen •  To create a new job use new JobManager.addJob() API with the topic chosen

and the job payload •  Add offloading payload to job payload •  Bundle and deploy JobConsumer on topology instances •  Enable/Disable the new topic on the instances, using Offloading Browser

DEVELOPMENT - RECIPE

62

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

63

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

org.apache.sling.discovery.api •  Discovers topology •  Cross-Cluster detection •  Foundation for job distribution

64

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

org.apache.sling.event •  Uses Sling Discovery •  New JobConsumer API and job topics •  New JobManager API for creating new distributed jobs •  Distributes jobs based on available job topics and job queue

configuration. •  Distributes in the whole topology, including clustered and non-

clustered instances •  Can execute cluster local jobs only

65

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

com.adobe.granite.offloading.core

•  Builds on top of Sling Distributed Jobs •  Does not perform distribution •  Detects jobs distributed to non-clustered instances •  Transports the jobs and payload to non-clustered

instances •  Uses replication for transport •  Does not execute jobs

66

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

com.adobe.granite.workflow.core

•  Defines a job consumer to distribute the execution of whole workflows

•  Defines topic com/adobe/granite/workflow/offloading

•  Implements WorkflowOffloadingJobConsumer component

•  Supports clustered and non clustered installations

67

TAKE AWAY

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution

cq-dam-core

•  Makes use of com/adobe/granite/workflow/offloading topic from Workflow Distribution

•  New workflow step (external step) that creates a new job on topic com/adobe/granite/workflow/offloading

•  New “DAM Update Asset Offloading” workflow •  Supports clustered and non clustered

configurations

68

TAKE AWAY Potential Future

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

69


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

New OSGi Specification • Distributed Eventing • Cloud Computing

70


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

Job Distribution •  Improved load balancing • Pull based distribution

71


Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

Offloading • As part of job distribution • Even simpler setup

72

TAKE AWAY Questions?

Discovery


DAM Ingestion

Sling

Granite

AEM

Job Distribution


Fro

m E

ven

ting

to

Jo

b P

roc

ess

ing

73

Adobe AEM - From Eventing to Job Processing

Technology

instance sling id optional

single instance instance

unique id sling id cluster

sling job manager

unique id sling id topologies

apache sling discovery

job consumer

job note