Feb-16WIJIS Messaging Model G. W. Pasdirtz, Ph.D., WIJIS Joe Mierwa, Visionair.

05/08/23 WIJIS Messaging Model

WIJIS Messaging Model

G. W. Pasdirtz, Ph.D., WIJISJoe Mierwa, Visionair


AgendaHigh Level OverviewAssociations

Current mechanismsProposed SolutionsA WIJIS solution

GJXDM Object-Relational Mapping IssuesObject Inclusion vs. Object SeparationObject ReferenceAggregation vs. CompositionAssociations


The Context


WIJIS Federated SOA

LDAP

Data

ha svxuydyc jhvjd lm qp dc pdp io dld fk cpoc pe oc jp eo c j;d flce ic ine oicn oe ie eiknc frn oe rno e rio eripe qriiero ie ro iero froerino iqne ro in ero ine rbd ikd bq w iqwu b iu bied jck jb c ju cb ie jb ce jbcjck jcb kjb ejbcke jcbeje bkjqw b ckq b w wc jcwejbk jw b kqw jb kwjblw jken lw e n lq w e knd

Portal

LDAP

Data

ha svxuyd ycjhv jd lm q pd cp dpiodld fk cpo cp eo c jp eo cj;d flce icine o icn oe ie e iknc frnoe rn oe rio e ripe q riiero ie roie ro froerin o iqn ero in e ro ine rbd ikd bq wiqw ub iu bied jck jbcjuc bie jb ce jb cjck jcb k jbe jb ckejcbeje b kjq w bckqb ww c jcw e jb k jwb kq w jb kw jblw jken lw en lq we kn d

Portal

LDAP

Data

h asvxuyd yc jhvjd lm q pd cp dpiodldfk cpo cp eo cjpe oc j;d f lce ic in eo icn oe iee iknc frno ern oe rioe rip eq riiero ie ro ie rofroe rin oiqn ero in e roin erb dikd b qw iq w ub iu b ied jck jbc jucbiejb ce jb cjck jcb kjbe jb ckejcb ejeb kjq w bc kqb w wc jcwe jb k jw b kq w jb kw jblw jke nlw en lq w e kn d

Portal

LDAP

Data

ha svxuydyc jhvjd lm qp d cpdp io dld fk cpoc pe oc jp eo c j;d flce ic ine oicn oe ie eiknc frn oe rno e rio eripe qri iero ie ro iero froerino iqne ro in ero ine rbd ikd bq w iqw u b iu bied jck jb c jucb ie jb ce jbcjckjcb k jb ejbcke jc beje bkjqw b ckq b w w cjcw ejbk jw b kqw jb kwjblw jken lw e n lq w e knd

Portal

LawEnforcement

W IJISSearch

EnterpriseService

BusDOJ DA Corrections

Index

Agency w/oWeb

Services

Data

Web Services (W S, W SRP, W S-Security, SOAP, SAML)

hasvxuydycjhvjdlm qpdcpdpiodldfkcpocpeoc jpeocj;dflce ic ineoicn oeiee ikncfrnoernoerioeripeqriiero ieroierofroerino iqn ero ineroinerbdikdbqwiqwub iub iedjckjbcju cbiejbcejbcjckjcbkjbe jbckejcbejebkjqwbckqbwwc jcwejbkjwbkqwjbkw jblwjke nlwe nlqwekn d

EnterprisePortal

LDAP


Workflow


The WIJIS IndexRelies on a Publish/Subscribe modelDoes NOT extract data from ODSs or Data WarehousesIs NOT a data mart

It is a pointer system optimized for searching


GJXDM & NIEM

NIEM 1.0

HomelandSecurity

Other

Justice

GJXDM 3.X Core

U-Core


GJXDM Shallow Objects


Multiple Message Models

XML-DOC: incident, arrest, citation, etc.XML-RPC: queryXML-DB: publish to the WIJS indexFlat File: SOA Class C Systems


Object-Relational Mapping

XML Schema(XSD)

Object Schema(UML)

RDB Schema(SQL)

XML Data Binding Object-RelationalMapping


The Problem


Problem StatementExisting GJXDM mechanisms are too simplistic to reflect information kept in the Index

Don’t handle reflexive relationships very wellDon’t translate very well from objects into database entries


Existing Mechanism #1XML-DOC Object Inclusion

Also referred to as hierarchical inclusion

Activity

“Investigator” “Bob”

ActivityWitness

“Bob”“I saw the whole thing”

WitnessAccountDescriptionTextPersonName PersonNameActivityOfficial.Enforcement

Bob2

(EnforcementOfficialType)Bob1

(WitnessType)Same-As

ActivityOfficial.Enforcement


Example: Object Inclusion Instance <WIJIS namespace stuff goes here

xsi:schemaLocation="http://schemapathdefinition/WijisIndex.xsd"><Activity j:id="ActID1">

<ActivityID>4-00-09796-0399-181</jdd:ActivityID><ActivityDate>2000-08-23</jdd:ActivityDate><ActivityEndDate>14:31:00</jdd:ActivityEndDate><ActivityOfficial.Enforcement ID=“PID1”>

<PersonName><PersonFullName>Bob</PersonFullName></PersonName><EnforcementOfficialTypeText>some text </EnforcementOfficialTypeText><EnforcementOfficialBadgeID>

<ID>1234</ID><EnforcenmentOfficialBadgeID><EnforcementOfficialUnit ID="OID1" >

<OrganizationID><ID>1234</ID></OrganizationID><OrganizationName>NameText</OrganizationName>

</EnforcementOfficialUnit></ActivityOfficialEnforcement<j:ActivityWitness ID=“PID1”>

<PersonName><PersonFullName>Bob</PersonFullName></PersonName><WitnessID></WitnessID><WitnessAccountDescriptionText></WitnessAccountDescriptionText><WitnessDate></WitnessDate>

</j:ActivityWitness></Activity>

</WIJIS>

Objects are includedwithin the Activity. ID value indicates they are the same person.


Existing Mechanism #2Use of references

Also referred to as Aggregation

+Operations()-Attributes

Person

+operations()-attributes

PersonMarriagePartner

+Operations()-Attributes

PersonAccompliceWithSubject


PersonSocialDetails


PersonAssignedIDDetails


PersonBiometricDetails


PersonMedicalDetails

111111


Example: References Instance<WIJIS namespace stuff goes here

xsi:schemaLocation="http://schemapathdefinition/WijisIndex.xsd"><Activity j:id="ActID1">

<ActivityID>4-00-09796-0399-181</jdd:ActivityID><ActivityDate>2000-08-23</jdd:ActivityDate><ActivityEndDate>14:31:00</jdd:ActivityEndDate><ActivityOfficialReference.Enforcement ref=“PID1”/><j:ActivityWitnessReference ref=“PID2” />

</Activity><EnforcementOfficial ID=“PID1”>

<PersonName><PersonFullName>Bob</PersonFullName></PersonName><EnforcementOfficialTypeText>some text </EnforcementOfficialTypeText><EnforcementOfficialBadgeID><ID>1234</ID></EnforcenmentOfficialBadgeID><EnforcementOfficialUnit ID="OID1" >

<OrganizationID><ID>1234</ID></OrganizationID></EnforcementOfficialUnit>

</EnforcementOfficial><j:Witness ID=“PID2”>

<PersonName><PersonFullName>Bob</PersonFullName></PersonName><WitnessID></WitnessID><WitnessAccountDescriptionText></WitnessAccountDescriptionText><WitnessDate></WitnessDate>

</j:Witness></WIJIS>

Objects are referencedfrom the Activity.


Proposed Solutions


Object SeparationRole.Of (Georgia Tech)XML-DB

Single Association Table (NDEx)


Role.Of UMLSuperType

WitnessType PersonType EnforcementOfficialType

1

-Role-Of 1

-Role-Of


Role.Of Data GraphActivity

“Investigator”

“Bob”

ActivityWitness

“I saw the whole thing”

WitnessAccountDescriptionText

PersonName


Bob2

(EnforcementOfficialType)

Bob1

(WitnessType)

Role-Of


Bob3

(PersonType)

Role-Of


Role.Of Instance<WIJIS namespace stuff goes here

xsi:schemaLocation="http://schemapathdefinition/WijisIndex.xsd"><j:Activity>

<j:ActivityWitness j:id="b1"> <j:RoleOf.Person j:id="p1">

<j:PersonName> <j:PersonGivenName>Bob</j:PersonGivenName>

</j:PersonName> </j:RoleOf.Person><j:WitnessAccountDescriptionText>

I saw the whole thing</j:WitnessAccountDescriptionText>

</j:ActivityWitness> <j:ActivityOfficial.Enforcement j:id="b2">

<j:RoleOfReference.Person j:ref="p1"/> <j:EnforcementOfficialTypeText>Investigator</j:EnforcementOfficialTypeText></j:ActivityOfficial.Enforcement>

</j:Activity><WIJIS>


Role-Of IssuesDoes not handle all possible associations between objects.

E.g. GJXDM field EmploymentEmployer could be modeled as RoleOf.Employer. However

EmploymentAssignedUnit.Organization, describing association between person object & assigned unit in an organization

Not a good fit for Role.Of modelingNor does EmploymentLocation

An association between person and organization location


NDEx ApproachNDEx proposes to handle missing associations in GJXDM via:

Single Association TableCodes

AssociationCodeOrganizationTypeIncludes codes for

Unknown, Other, None, Affiliate, Contractor, & Sub-contractor.AssociationCodeOrganizationLocation

Include codes for Unknown, Other, None, Physical Street Address, Mailing Address and Billing Address.

AssociationCodeOrganizationPersonIncludes codes for Unknown, Other, None, Owner, Full-time Employee, Part-time Employee, Contractor and Member.

Etc…Approach is a hybrid

IncidentVictim & IncidentWitness handled via references instead of codes describing association of person with incident.


Single Association Table UML

NDEx proposes a single association

table using references & codes


Single Association Table Instance

<i:Association j:effectiveDate="1999-07-15">

<i:AssociationSubjectReference j:ref="PID1"/><i:AssociationObjectReference j:ref="LID1"/><i:AssociationCode.PersonLocation>

3</i:AssociationCode.PersonLocation>

</i:Association>


QuestionCan GJXDM constructs such as IncidentVictim & IncidentWitness be eliminated in favor of association codes?

Would lose data related to association WitnessAccountDescriptionText field.

More modeling could handle this problem Document object with PersonActivity association coded as “statement”.


Associations - IIn theory, many possible associations between GJXDM objects exist

For indexing, most important associations derive from relationships to activity

ActivityPerson (victim, witness),ActivityProperty (seized property) ActivityOrganization (ActivityClearer.Organization)ActivityDocument (incident report)ActivityLocation (ActivityLocation)Associated GJXDM attributes are in parentheses.


Associations - IIThere are also many direct relationships of interest

PersonOrganization (EmploymentEmployer)OrganizationPerson (OrganizationPrincipalOfficial)PersonContactInfromation (PersonDayContact)OrganizationContactInformation (OrganizationDayContact)PersonLocation (EmploymentLocation)OrganizationLocation (OrganizationLocation)PropertyPerson (PropertyTitleHolder.Person)PropertyLocation (PropertyLocation)


Associations - IIIThere are also relationships between the same object

Referred to as Reflexive assocationsPersonPerson (PersonAlias)OrganizationOrganization (OrganizationSubunit)DocumentDocument (document correction)ActivityActivity (previous incident).


Associations


The WIJIS Approach


WIJIS SolutionExtend GJXDM to handle object relationships using multiple association tables

One for each of the associated objectsFacilitates mapping for indexing databasesSemantics of the associations are more easily modeled


Multiple Association Tables


Schema Snippet - I…<xs:complexType name="AssociationType">

<xs:attribute ref="j:ref"/></xs:complexType><xs:complexType name="ActivityActivityType">

<xs:complexContent><xs:extension base="ext:AssociationType">

<xs:attribute name="code" type="codes:ActivityActivityCodeType" default="none"/>

</xs:extension></xs:complexContent>

</xs:complexType>


Schema Snippet - II<xs:simpleType name="ActivityActivityCodeType">

<xs:restriction base="xs:token"><xs:enumeration value="unknown"/><xs:enumeration value="other"/><xs:enumeration value="none"/><xs:enumeration value="activity_previous_incident"/><xs:enumeration value="activity_related_incident"/><xs:enumeration value="activity_previous_court"/><xs:enumeration value="activity_related_case"/><xs:enumeration value="activity_related_activity"/>

</xs:restriction></xs:simpleType>


Schema Snippet - III<xs:complexType name="ActivityType">

<xs:complexContent><xs:extension base="j:ActivityType">

<xs:sequence><xs:element ref="ext:ActivityActivity" /><xs:element ref="ext:ActivityDocument“ /><xs:element ref="ext:ActivityLocation" /><xs:element ref="ext:ActivityOrganization" /><xs:element ref="ext:ActivityPerson" /><xs:element ref="ext:ActivityProperty" />

</xs:sequence></xs:extension>

</xs:complexContent></xs:complexType>


XML-DB Instance<WIJIS >

<Activity id="ACTID1“><ActivityPerson ref=“PID1” code=“witness”/><ActivityPerson ref=“PID1” code=“enforcement_official”/>

</Activity><Person id="PID1" >

<PersonName><PersonFullName>Bob</PersonFullName>

</PersonName><PersonOrganization ref=“OID1” code=enforcement_official”/>

</Person><Organization id="OID1>

<OrganizationORID><ID>1234</ID></OrganizationORID><OrganizationName>Local PD</OrganizationName>

</Organization></WIJIS>


Object-Relational MapVery straightforwardCREATE TABLE WIJIS (

WIJISID INTEGER PRIMARY KEY,reportingOrganizationText VARCHAR(124) NOT NULL,reportedDate DATE NOT NULL,reportedTime TIME NOT NULL,sourceIDText VARCHAR(124) NULL,sensitivityText VARCHAR(124) NOT NULL,commentText VARCHAR(124) NULL,distributionText VARCHAR(124),RECEIVEDDATE DATE NOT NULL,RECEIVEDTIME TIME NOT NULL

)CREATE TABLE ACTIVITY (

ACTID INTEGER PRIMARY KEY,WIJISID INTEGER NOT NULL,ActivityID VARCHAR(12) NOT NULL,ActivityDate DATE NOT NULL,ActivityEndDate DATE NULL,FORIEGN KEY (WIJISID) REFERENCES WIJIS (WIJISID))

Each object has table with Primary & Foreign

key related to parent object


Association Mapping

<!ELEMENT ActivityActivity EMPTY > <!ATTLIST ActivityActivity ref CDATA #REQUIRED code (unknown|other|none|previous_incident|related_incident| previous_court|related_case|related_activity) "none">

CREATE TABLE ACTIVITYACTIVITY ( CODE VARCHAR(24) NOT NULL, ACTAID INTEGER PRIMARY KEY, ACTID1 INTEGER NOT NULL, ACTID2 INTEGER NOT NULL, FOREIGN KEY (ACTID1) REFERENCES ACTIVITY (ACTID), FOREIGN KEY (ACTID2) REFERENCES ACTIVITY (PERID)

)


Association MappingMapped in a similar manner

Have 2 foreign keys

CREATE TABLE ACTIVITYACTIVITY (CODE VARCHAR(24) NOT NULL,ACTAID INTEGER PRIMARY KEY,ACTID1 INTEGER NOT NULL,ACTID2 INTEGER NOT NULL,FOREIGN KEY (ACTID1) REFERENCES ACTIVITY (ACTID),FOREIGN KEY (ACTID2) REFERENCES ACTIVITY (PERID) )


XML-RPC InstanceFor queries, association codes are used for searching

<?xml version="1.0" encoding="UTF-8"?><WIJIS_PersonSearch> <Person PersonName=”Martha Stewart" > <ActivityPerson code="arrest_subject"/> </Person></WIJIS_PersonSearch>For queries,


Has It Become Too Complicated?


Alternatives


Does It Matter?


Relational DB StorageUnstructured Storage

Entire XML document stored entirely as Character Large Object (CLOB).

Providesrapid throughputdocument fidelityflexibility.

Document processing can be expensive Document Object Model (DOM)-level operations are required.


Relational DB StorageStructured Storage

Provides many advantagesOptimized memory managementXquery and Xpath capabilitiesReduced storage requirementsB-tree indexing and in-place updates.

Schema annotations used to control storage of XML collections:

CLOB - XML text stored in a Character Large Object.VARRAY in a LOB - Each element converted into database-native object.VARRAY as Nested Table

Each element converted into database native objectStored as set of rows in Index Organized Table (IOT).

VARRAY of REF XMLType as LOB - Each element treated as separate CMLType value.VARRAY of REF XML Type as Nested Table - Each element treated as separate CMLType value stored as Nested Table


Questions?

Feb-16WIJIS Messaging Model G. W. Pasdirtz, Ph.D., WIJIS Joe Mierwa, Visionair.

Documents