Signals Intelligence Alert & Prediction System (SINAPS ...

APPROVED FOR PUBLIC RELEASE


U.S. ARMY COMBAT CAPABILITIES DEVELOPMENT COMMAND

AVIATION & MISSILE CENTER

06 AUGUST 2019

Signals Intelligence Alert & Prediction System (SINAPS) Artificial Intelligence Enhancements to SIGINT Analysis

T. Warren de Wit, Radiance Technologies

Software Engineer, SINAPS Technical Lead

Daniel Anderson, Radiance Technologies

AVP Stennis Operations; SINAPS PM

Andrew Zinn, Hill Technical Solutions, Inc.

SINAPS Program Manager

Jim Buford, Branch Chief

Systems Simulation, Software, and Integration;

Advanced Technology

Mark Umansky, SINAPS Lead

Systems Simulation, Software, and Integration;

Advanced Technology

Distribution Statement A: Approved for public

release. Distribution is unlimited.

2



CCDC AvMC – Intelligence Model and Simulation Cell (IMSC)

• Current & Future Intel Systems

• Conduct Feasibility Analysis

Signals Intelligence Alert and Prediction System (SINAPS)

• Artificial Intelligence (AI) & Machine Learning (ML)

• SIGINT Exploitation, Dissemination, Feedback

• Two Algorithms – Anomaly Detection & Threat Prediction

BACKGROUND

3



IDENTIFICATION OF THE PROBLEM

Complex environment Limited time, resources

High, increasing data

Volume, Velocity

Obscure data of

significanceUnanalyzed & Irrelevant-Looking Data needs Automated Processes

4



SINAPS VALUE

Uncollected

SIGINT

Processed

Collected SIGINT

Perished

Collected SIGINT

100% Evaluated or

Processed

Current Estimates

SINAPS

SINAPS can process & evaluate all collected data using AI/ML

5



ANALYSTS

SINAPS COMPONENTS - OPERATIONAL

Collected SIGINT SINAPS

Email Alerts

(Current)

IRC

Notifications

(Future)

Dashboard

(Future)

Map Updates

(KML)

(Future)

SIGINT Data

Sensor Data

Analyst Reports

Anomalies & Potential Threats

are Identified

Collection Tasking Can be Informed and Prioritized Based on SINAPS-Produced Alerts

Analysts Can Prioritize Activities through SINAPS-Produced Alerts

6



SINAPS – USER SETUP

User sets geographic area

of interest.

User sets minimum

confidence level for alerts.

Analysts Subscribe to SINAPS Alerts

7



SINAPS – ALERTS

8/20/2004 4:00:00 PM -06:00

Sample Anomaly Detection Alert

Sample Predictive Alert

8



SINAPS – END-TO-END

9



Anomalous activity

has been detected

on a frequency of

XXX.XXX MHz.

Unnatural

pattern

• Receive notifications of anomalous data

• Serves as an early threat warning system for activity that

may have been discovered later or not at all

• Notifications can drive collection tasking and analysis

• Constantly learning as the environment evolves

• Continuously monitor the entire collection environment

SOLUTION - ANOMALY DETECTION

10



ALGORITHMS: ANOMALY DETECTION

• Anomalous attributes can be retraced

instead of being a black box

• Numeric matrix representation conducive to

parallel and/or distributed implementation

• Can determine multiple types of anomalies

• No data-specific dependencies

• Reduced data complexity

• Provides general picture of data

• Computed per designated time intervals

• Summary statistics are interchangeable

• Computationally fast

• Compare data to itself

• No a priori information required

• Will readily transition to other scenarios

Algorithm Overview

• Divide data into time intervals

• Compute summary variables for

each interval

• Text scoring using SVM is a variable

• For each interval, compute outlier

values across frequencies for each

variable.

• For each frequency, compute outlier

values across time for each

variable.

• Use the resulting outlier matrix

(0/1’s) to detect outlier-of-outlier

frequencies and variables across

time and frequency based on counts

Algorithm Strengths

Multi-Layer, Cross-Cut Anomaly Detection

11



SINAPS has

predicted an

IED attack

within the next

7 days

SOLUTION - PREDICTION

• Machine Learning Based (Random Forest)

• Subscribe to regional notifications

• Receive predictions of SIGACTs up to 4 weeks in advance

• Understand the geographic probability of the predicted event

• Receive updates to watch as confidence changes with time

12



ALGORITHMS: PREDICTION

Algorithm Overview

• Actually two random forest algorithms

trained for long-term and short-term

predictions

• Tunable parameters adjust for processing

speed vs. predictive performance

• Text scoring using SVM is a variable

• Produces a prediction of time until event

• Issues a confidence rating

Random Forest Prediction Algorithm

13



MODELING AND SIMULATION APPROACH

FlatMap SIGINT Environment Simulator

– Models a realistic signal environment; Creates a large dataset of routine

communications

– Simulates typical emitting systems such as police, emergency departments, taxi

companies, etc.

– SIGACT scenarios are injected randomly into the signal environment. IED placement

used for prototyping.

The generated data includes• Metadata

• Lines of Position (LOPs): ~10 million

records

• Ellipses: ~1 million records

• Analyst reports: ~10 thousand records

• SIGACT scenarios:

• Overlaps

• Red Herring

14



SIGACT SCENARIO GENERATION

CL>FCL>TM

CL>BM

BM>S1

TM>S2

BM>S1

TM>S2BM>TM BM>TM BM>E

R2>E

R>ER>E R2>E

OD>CM

R1

E

E>OD

CM

L>OD R2>OD

L>OD

R1>OD

L>OD

OD>R OD>DEVIC

E

L>OD

OD>CL CM>CL CL>F

Money Plans,

Money

Order Delivery Coord. Bomb

Test

Bomb

Delivery

Plan

Coord

DetonatePlan

Coord

Plan

Coord

SIGACT

Report

ROIFinal Plan

TIMELINE

T>E

• Actors– Cell Leader (CL)

– Financier (F)

– Trigger Maker (TM)

– Observer/Detonator (OD)

– Bomb Maker (BM)

– Supplier (S1), Supplier (S2)

– Transporter (T)

– Emplacer (E)

– Recon (R1), Recon (R2)

– Camera Man (CM)

– Lookout (L) – person located near the Detonator (OD)

D-.001D-30

days

D-29

days

D-27

days

D-20

days

D-13

days

D-12

days

D-10

days

D-8

days

D-5

days

D-3

days

D-1

day

D-.1D-8

hrs

D+24

hrs

D+48

hrs

D!! D+.5D-10

hrs

E, T

R1

CM

OD

300m

3KM

10KMTM

R2

S1

30KMBM

CL

F, Another City

S2

L

15



PREDICTION PROCESS DIAGRAM

16



ALGORITHM PERFORMANCE

17



ANOMALY DETECTION PERFORMANCE

• Most anomaly detections are associated with SIGACTs or the intentional benign

event (Red Herring).

• 100% of nefarious scenarios were identified as anomalous

• Only 2% of data flagged as anomalous was from the background noise dataset

0 500 1000 1500 2000 2500 3000 3500 4000

Nefarious

Benign Event

Benign Noise

Tru

th

Nefarious Benign Event Benign Noise

Nefarious 2984 553 0

Unknown 3632 903 70

Benign 7 0 82

SIGACT Truth vs. SINAPS Detection and Rating

18



PREDICTION PERFORMANCE

SINAPS Predictions Assessed on Limited Simulated Data.

<=4wk <=3wk <=2wk <=1wk Active

Benign 84 6 0 3 3

Nefarious 87 56 14 29 41

0

20

40

60

80

100

120

140

160

180

Axi

s Ti

tle

When Predictions Occur

Nefarious Benign

0 20 40 60 80 100 120

Nefarious

Benign

SINAPS Prediction and Rating

Tru

th

Nefarious Benign

Very High 97 9

High 85 8

Moderate 13 43

Low 32 36

Event Truth vs. SINAPS Prediction and Rating

Very High High Moderate Low

• Benign (false positive) predictions are

more likely to be rated as a low or

moderate risk.

• Correct predictions are more likely to be

labeled high or very high risk.

• Most false positive predictions (8.3%)

are long term predictions.

• Most short-term predictions are

associated with SIGACTs.

19



• SINAPS Benefits

– It is GOTS

– Promotes evaluation of all collected SIGINT data

– Increases situational awareness

– Approach is data independent and can detect numerous types of anomalies

– Highly modular – add new techniques with minimal development effort.

– Improves collection tasking efficiency and effectiveness

– Alerts analysts to anomalous data and predicted SIGACTs

– Assists analysts with data correlation

– Provides analytical continuity for all units

CONCLUSION

The enduring benefit is the ability to recognize and alert to anomalous activity and

issue advanced predictions with high confidence for SIGACTS that represent a

threat to U.S. Armed Forces or their interests.

20



Web Site

www.amrdec.army.mil

Facebook

www.facebook.com/ccdc.avm

Instagram

www.instagram.com/CCDC_AVM

Twitter

@CCDC_AVM

Public Affairs

[email protected]

Signals Intelligence Alert & Prediction System (SINAPS ...

Documents