Organizing the world of CQ rest infinitive possibilities by Arkadiusz Kita

Organizing The Worldof

Adobe CQ REST infinitive possibilities

[email protected]

9th April 2014

Arkadiusz Kita CQ Developer

Wednesday morning, 9th April 2014...

...another beautiful day at work!

There is time for my favouritecoffee!

Wednesday morning, 9th April 2014...

...another beautiful day at work!

...but wait - there is an importante-mail in my inbox!

1 The client needs PoC ASAP

2 Next week 1 mln hits campaign!

3 Agency is ready for content entry!

Mainframe

4 Ad-hoc infrastructure!

5 Monitoring/maintenance required!




Mainframe






Mainframe






Mainframe






Mainframe



It seems like huge pressure and a lot of work to do...

...or maybe rather a lot of CQ clicks!

Managing a cloud of...

...thousands of AEM/CQ instances

...hundreds of dev, test, qa, uat, live environments...tens of client deployments

in a consistent way!


...thousands of AEM/CQ instances...hundreds of dev, test, qa, uat, live environments

...tens of client deployments










Providing condensed...

checklists knowledge know-how

...for developers/administrators

in a reusable way!


checklists

knowledge know-how


in a reusable way!


checklists knowledge

know-how


in a reusable way!




in a reusable way!




in a reusable way!

Last but not least!

1 Automating repetetive tasks

2 Improving safety and security by not touching anything twice3 Focusing on real challenges!!! Discovering new possibilities!4 Being happy, not bored!5 Loving what you do!

Last but not least!

1 Automating repetetive tasks2 Improving safety and security by not touching anything twice

3 Focusing on real challenges!!! Discovering new possibilities!4 Being happy, not bored!5 Loving what you do!

Last but not least!

1 Automating repetetive tasks2 Improving safety and security by not touching anything twice3 Focusing on real challenges!!! Discovering new possibilities!

4 Being happy, not bored!5 Loving what you do!

Last but not least!

1 Automating repetetive tasks2 Improving safety and security by not touching anything twice3 Focusing on real challenges!!! Discovering new possibilities!4 Being happy, not bored!

5 Loving what you do!

Last but not least!

1 Automating repetetive tasks2 Improving safety and security by not touching anything twice3 Focusing on real challenges!!! Discovering new possibilities!4 Being happy, not bored!5 Loving what you do!

Use AEM/CQ REST infinitive possibilities!

UNIXUNIXTry the github.com/Cognifide/CQ-Unix-Toolkit!

Start using it!Help us develop it!

https://github.com/Cognifide/CQ-Unix-Toolkit

What is the CQ Unix Toolkit?

+ + = UNIXUNIX

open source user-friendly curl wrapper

set of independent simple micro tools

AEM/CQ HTTP REST semi-code/semi-documentation

allows to control CQ/dispatcher by HTTP

Commons are extracted into one cqapi tool.


+ + = UNIXUNIX







+ + = UNIXUNIX







+ + = UNIXUNIX







+ + = UNIXUNIX







+ + = UNIXUNIX






Use cases

Case 1 — distributed REST knowledge

No common place for AEM/CQ REST documentation!

Case 2 — hacking packages

1 Installing hotfixes/releases automaticallycqput; cqrun

2 Synchronizing content between instancescqpkg; cqput; cqbld; cqcp; cqrun; cqsnp

3 Making path-based backupscqpkg; cqput; cqbld; cqcp

4 Uninstalling deleted packagescqsnp

5 Clearing content pathscqpkg; cqput; cqrun

6 Moving paths from repository externallycqpkg; cqput; cqrun; cqsnp; cqsnp

7 Deleting unknown/garbage packagescqls; cqdel; cqsnp

























































Case 3 — monitoring/maintenance

1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring

2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status

3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks

4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction

5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge

6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection

7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool

8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)

$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ


1 cqmon – Sling JMX monitoring2 cqwfl – running/stale workflows status3 cqchk – repository consistency checks4 cqtpm – TarPM compaction5 cqmrg – Lucene indexes merge6 cqgc – DataStore garbage collection7 cqdam – AEM/CQ DAM Health Tool8 cqosgi + diff – bundle states (by using patterns)$ cqosgi -m > pattern-file;$ cqosgi -s org.apache.sling.jcr.davexStopping bundle: org.apache.sling.jcr.davex (#109)$ cqosgi -m | diff -q - pattern-fileFiles - and pattern-file differ

Case 4 — common devops tasks

1 cqosgi – automated bundle management2 cqpkg – create ad-hoc empty packages3 cqrepkg – fast offline repackaging for your content snapshots4 cqclr – clear dispatcher cache5 cqcfg – change bundle configuration remotely

configure SMTP clienttune Sling Main servlet

6 cqjcr – list JCR nodes, change and add themdynamically-scoped backups:cqjcr; cqpkg; cqbld; cqcp; cqdelchange Sling redirect/disable ContentFinder


1 cqosgi – automated bundle management

2 cqpkg – create ad-hoc empty packages3 cqrepkg – fast offline repackaging for your content snapshots4 cqclr – clear dispatcher cache5 cqcfg – change bundle configuration remotely




1 cqosgi – automated bundle management2 cqpkg – create ad-hoc empty packages

3 cqrepkg – fast offline repackaging for your content snapshots4 cqclr – clear dispatcher cache5 cqcfg – change bundle configuration remotely




1 cqosgi – automated bundle management2 cqpkg – create ad-hoc empty packages3 cqrepkg – fast offline repackaging for your content snapshots

4 cqclr – clear dispatcher cache5 cqcfg – change bundle configuration remotely




1 cqosgi – automated bundle management2 cqpkg – create ad-hoc empty packages3 cqrepkg – fast offline repackaging for your content snapshots4 cqclr – clear dispatcher cache

5 cqcfg – change bundle configuration remotelyconfigure SMTP clienttune Sling Main servlet










Development

Create new tools with us!Organize the AEM/CQ REST World!

DevelopmentCreate new tools with us!

Organize the AEM/CQ REST World!

Rule 1 — Capture day to day requests

Rule 2 — Write user-friendly usage messages

$ cqbldUsage: cqbld [OPTION...] package-nameBuild (rebuild) already uploaded package by group id and name in CQ PackageManager using instance URL. Examples: cqbld -u admin pack # Build package named pack cqbld -u admin -g GRP pack # Build package named pack in group GRP cqbld -i http://localhost:5510 # Build package for localhost instance on tcp -g com.group stuff # port 5510 named stuff in group:com.group -p secret # with password provided: secret

Options:

-u use specified usernamed for connection -p use provided password for authentication -i use specified instance URL to connect -g locate package by additional group ID

Rule 3 — Choose descriptive code over comments

# Invoke curl tool with current referer and # instance URL and credentials to get list # of packages STATUS=`$CURLBIN -s --write-out "%{http_code}"\ -u "$u:$p" -H "Referer: $url/crx/packmgr"\ -F cmd=ls "$url/crx/packmgr/service.jsp"`


# Invoke curl tool with current referer and # instance URL and credentials to get list # of packages STATUS=`$CURLBIN -s --write-out "%{http_code}"\ -u "$u:$p" -H "Referer: $url/crx/packmgr"\ -F cmd=ls "$url/crx/packmgr/service.jsp"`


PACKAGE_MANAGER_PATH="/crx/packmgr"REFERER="${instance}${PACKAGE_MANAGER_PATH}" AUTH="${username}:${password}" FILEPATH="${PACKAGE_MANAGER_PATH}/service.jsp" FULL_URL="${instance}${FILEPATH}" REFERER_HEADER= "Referer: ${REFERER}" COMMAND="cmd=ls"

Use shellcheck.net to validate POSIX shell syntax!

http://www.shellcheck.net/


STATUS=$("${CURLBIN}" \ -s \ --write-out "%{http_code}" \ -u "${AUTH}" \ -H "${REFERER_HEADER}" \ -F "${COMMAND}" \ "${FULL_URL}")


STATUS=$("${CURLBIN}" \ -s \ --write-out "%{http_code}" \ -u "${AUTH}" \ -H "${REFERER_HEADER}" \ -F "${COMMAND}" \ "${FULL_URL}")

Rule 4 — Use POSIX getopts for custom parameters

while getopts ":abc${apigetopts}" optdo case "${opt}" in

...

\?) echo "Invalid option: -${OPTARG}" >&2; _usage;; :) echo "Option -${OPTARG} requires an argument." >&2;

_usage;; esac done

Rule 5 — Use cqapi to detect/handle curl command

cqapi

shared code, status, parsing common options, error handling

Rule 6 — Use the most usable REST HTTP

pick!

rate!

choose!

Rule 7 — Reserve API cmdline options for CQ/AEM connectivity

-u username-i instance URL-p password

Rule 8 — Avoid including external custom dependencies

This can be harmful,hard in use and notportable!

Rule 9 — Keep It Simple Stupid

KiSS

Keep toolkit filesas small as possibleup to 400 LOC

Manage complexity!Simplify logic!

Rule 9 — Keep It Simple Stupid

KiSS

Keep toolkit filesas small as possibleup to 400 LOC

Manage complexity!Simplify logic!

Thank you!

Q&A

Resources


Presentation symbols provided by Tango Desktop Project

blog entry on Cognifide website


http://tango.freedesktop.org/Tango_Desktop_Project

http://www.cognifide.com/blogs/cq/cq-unix-toolkit/

Organizing the world of CQ rest infinitive possibilities by Arkadiusz Kita

Technology

simple stupid kiss

repository externally cqpkg

based backups cqpkg

cq unix toolkit

packages status curlbin

toolkit les

instances cqpkg

aem cq instances