Netglub : Really Open Source Information Gathering Guillaume PRIGENT - Founder/CTO diateam <[email protected]> This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License. HITBSECCONF2011 - May 17-20 - Amsterdam
Netglub : Really Open Source InformationGathering
Guillaume PRIGENT - Founder/CTO diateam
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
HITBSECCONF2011 - May 17-20 - Amsterdam
IntroductionOSINT process
NetglubPerspectives
Plan
1 Introduction
2 OSINT process
3 Netglub
4 Perspectives
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 2 / 30
IntroductionOSINT process
NetglubPerspectives
Disclaimer**a big fat one, because everybody loves fineprint
Information is provided as-is, though every effort has been made to ensure the accuracy of the informationpresented. Author of the presentation is not legally liable under any circumstances for any damages such asbut not limited to (including direct, indirect, incidental, special, consequential, exemplary or punitive damages)resulting from the use or application of the presented information.
Unless explicitly noted in forms such as but not limited to "the XYZ Company says", etc., the opinions ex-pressed in this presentation are solely and entirely my own.They should not be interpreted as representing the positions of any organization (past, present, future, exis-tent, non-existent, public, private, or otherwise) with which I may or may not have been, are or are not, or willor will not be affiliated at some time in the past, present, or future.
All trademarks and registered names are the property of their respective owners. All the effort has been madeto link to the original material used as exhibition items in the presentation, and those items are property oftheir respective owners.
This presentation is c©2010, Guillaume Prigent <[email protected]>. Released under :
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 3 / 30
IntroductionOSINT process
NetglubPerspectives
ContextDomainWhat for ?
ContextOr "What are we talking about ?"
DefinitionOSINT is a form of intelligence collection management that involvesfinding, selecting, and acquiring information from publicly availablesources and analyzing it to produce actionable intelligence...(FromWikipedia)
ConsiderationToday, OSIF = digital informationQuantity versus quality dilemmaIndependent of information’s "color"
Not to be confused !Open Source Information != Open Source Software
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 4 / 30
IntroductionOSINT process
NetglubPerspectives
ContextDomainWhat for ?
Domain mappingOr "How to link the real world and the digital world ?"
Real worldEmailsPersonsPhone numbersAddressesDocumentsPatents / ProjectsSentences / Words ...Habits / HobbiesSocial affinities...
Digital world
IP, hosts, netblocks, ASWhois records / rWhoisForward and reverse DNSGoogle fu / Deep webDocument metadataTwitter, Facebook, ...XFN, vCards, hCardsFace detection, ...ip2geo, Google Earth, ......
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 5 / 30
IntroductionOSINT process
NetglubPerspectives
ContextDomainWhat for ?
What for ?
Analyze « social networks » to try and predict the potentialof a commercial product ;Detect « key people » in a group or social network ;Use face detection to cross-reference profiling information ;Use social networks to obtain economic intelligence ;Agregate fiscal, administration, patent and shareholdinginformation for a « Tender offer » ;Perceive the technological environment of a product andidentify interdependencies ;Perform a technical and organizational reconnaissancebefore a penetration test ;...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 6 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Methodology 1/2Or "Open Source Intelligence cycle management"
Question ?
Source identification
Enrichment /Pruning
Acquisition /Extraction
Visualization /Interaction
Normalization /Analysis
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 7 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Methodology 2/2Or "Open Source Intelligence cycle management"
1 Identify the need (the question, on what and/or who) ;2 Identify the potential Open Sources for information
collection ;3 Acquire or extract the information elements in their context ;4 Analyze and normalize the actually useful information ;5 Visualize the produced dependencies and better
understand their structure ;6 Prune, and optionally qualify and enrich the results ;7 Repeat again...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 8 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Open Source selection...in most cases visible Web isn’t enough...
*Forgive me father, for i have sinned *I know
Pic
ture
co
urt
esy o
f P
.Ch
ap
pa
tte
- C
op
yrig
ht P
.Ch
ap
pa
tte
20
11
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 9 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Open Source selection...needles in a haystack...
Some sources
DNS, Whois, http://dnshistory.org, ...
http://www.netcraft.com, http://www.robtex.com, ...
http://www.ip2geo.com, http://www.maxmind.com, ...
http://www.infogreffe.com, http://www.societe.com, ...
http://fr.espacenet.com, http://www.inpi.com, ...
http://twitter.com, http://www.monster.com,
http://www.facebook.com, http://www.linkedin.com,
http://www.alchemyapi.com, http://www.opencalais.com, ...
...
,French DoD suppliershttp://www.achats.defense.gouv.fr/Annuaire-des-fournisseurs,13786
,French DoD contractshttps://www.achats.defense.gouv.fr/-Liste-des-marches-conclus,57343-
...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 10 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Transformation 1/5First chain
DNS Names IP Address Netblock ASDomains
TransformationDomain to DNS Names : MX/NS/Zone transfer/BruteforceDNS Names to IP Address : Resolve ,
IP Address to Netblock : WhoisNetblock to AS : Code routers, Web
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 11 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Transformation 2/5Forward chain
DNS Names IP Address Netblock ASDomains
TransformationDomain to DNS Names : MX/NS/Zone transfer/BruteforceDNS Names to IP Address : Resolve ,
IP Address to Netblock : WhoisNetblock to AS : Code routers, Web
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 12 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Transformation 3/5Six more transforms...via Whois
DNS Names IP Address Netblock ASDomains
Email Address Phone NumberPerson
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 13 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Transformation 4/5Six more transforms...using Search Engines, PGP servers, ...
Website
DNS Names IP Address Netblock ASDomains
Email Address Phone NumberPerson
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 14 / 30
IntroductionOSINT process
NetglubPerspectives
MethodologySources selectionTransformation
Transformation 5/5And more...
Website
DNS Names IP Address Netblock ASDomains
Phrase
Location
File Document Email Address Phone NumberSocial
Network Profile
Person
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 15 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
Maltego competitor ?@ROELOF : Ves mir SORTIR, a l�di v nem ZASRANCY
"...if you have been living under a rock"Visual tool (graph) for dealing with informationEntities and relationshipsPlatform for information integration & correlationhttp://www.paterva.com
Blackhat 2011, Blackhat 2009, Hack.lu 2008, CanSecWest 2007
Open Source ?Closed sourceApplication Service Provider / Software as a ServicePrivacy ? / Anonymity ?...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 16 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
Netglub’s design guidelines
Admiral William Studeman, 1992, USAThe plan establishes the goal of creating an integrated commu-nity open source architecture. The new architecture must pro-vide, among other things :
flexible collection,networked access to external data bases,immediate user and customer feedback, andautomated, profiled delivery of collected open sourceinformation...
...will be an Open Source Information Exchange comprising acentral switch and digital communications networks which inter-connect ...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 17 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
Development process...the uggly part...
Netglub lifecycle
analyze what exists & "state of the art" ;identifying COTS & APIs ;focus on Open Source ;identify the technological barriers ;define a flexible, scalable architecture ;implement various Proof of Concept (PoC) ;test & compare our results (benchmark) ;reiterate & (one day) complete the tool/framework.
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 18 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
General architectureNot quite a botnet...
MASTER
SLAVE 1 SLAVE 2 SLAVE i
OSIF DB 1 OSIF DB 2 OSIF DB jOSIF DB 3 OSIF DB j-1
<TLS><TLS>
<TLS>
USER k
<XML-RPC>
over <HTTPS> <XML-RPC>
over <HTTPS>
USER 1
Internet
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 19 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
Netglub’s componentsSlave(s), Master & GUI
Slave’s main featuresAutonomous network service(daemon)Login to "Master" /announcement"Job" runner & scheduler for"transforms"
GUI’s main featuresLogin to "Master" (XML-RPC)Relationships visualisation...
Master’s main featuresAutonomous network service(daemon)Command & Controlfederation of slavesAuthentications &Permissions for "Slaves" &Clients (GUI,CLI)"transforms" & "entities"databaseAPI XML-RPC for clients...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 20 / 30
IntroductionOSINT process
NetglubPerspectives
Our motivationsGeneral architectureComponentsDemonstration
Demonstration
Step 1/2 : NetglubUser featuresMore internal
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 21 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Perspectives
Work in progress
Sources selection & OntologiesPersonnal OSIF / Building datastoreQualification (community detection, confidence, timeattributes, ...)Reusability, ScalabilityAnonymityHuge graph layout and real-time interactionMaintainability, DistributionMachine learning...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 22 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Glubby...as a component for Netglub NG...
Motivations / NeedsReal time "force based" graph layout
Fast render in OpenGL for 3D & 2D
GPU and/or CPU based
Library & Open Source, ...
State of the artGraphViz (http://www.graphviz.org/)
Gephi (http://gephi.org/)
Tulip (http://tulip.labri.fr/)
Jung (http://jung.sourceforge.net/)
NetworkX (http://http://networkx.lanl.gov/)
Igraph (http://igraph.sourceforge.net/)
UbiGraph (http://ubietylab.net/ubigraph/)
...
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 23 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
What’s GPU ?Or "The return of the vector machine"
vector machine massively parallel Graphics Processing Unit
+ =
ProsFastCheapLow-powerFuture
ConsSpecializedHard to programBandwith problemsRapidly changing
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 24 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Limitations of GPUs...the dark side of the force...
If the GPU is so great, why are we still using the CPU ?
You can’t simply "port" existing code and algorithms !
Data-stream mindset requiredNot suitable to all problems
Pointer chasing impossible or inefficientRecursion
Debugging is hardHardware is designed without debug busDriver is closed
BottlenecksStandard API ?
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 25 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Network representation as graphGraph-theoretic data structures
*Vertices 5
*Edges
0 1
0 2
0 3
2 3
2 4
Edge list*
*Sometimes called incidence list
Connections
0 1 2 3 4
0
1
2
3
4
0 1 1 1 0
1 0 0 0 0
1 0 0 1 1
1 0 1 0 0
0 0 1 0 0
Adjacenty matrix4
1
32
0
0 1, 2, 3
1 0
2 0, 3, 4
3 0, 2
4 2
Adjacenty list
0 1 2 3 4
0
1
2
3
4
3 -1 -1 -1 0
-1 1 0 0 0
-1 0 3 -1 -1
-1 0 -1 2 0
0 0 -1 0 1
Laplacian matrix**Sometimes called admittance matrix
or Kirchhoff matrix
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 26 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Glubby internal graph representationGPU/CPU structure
4
1
32
0
3 0 1 3 3
1 2 3 0 0
4
3 4
2 7 1 7
0 2 2
0 1 2 3 4
12.2 -34.2 112.1 -4.27 -45.7 643.6 -34.6 -78.2 432.0 -99.6
12.2 -34.2 112.1 -4.27 -45.7 643.6 -34.6 -78.2 432.0 -99.6
float * A
300.0 100.0 300.0 200.0 100.0
float * B
float * Size
ulong * I
ulong * E
0 1 0 0 0int * Locked
int nbnode
int dimension
int k
float maxDisplace
float gravity
float speed
int noCollision
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 27 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
GlubbyOur implementation on CPU/GPU
Fruchterman-Reingold on GPU&CPUFruchterman, Thomas M. J. ; Reingold, Edward M. (1991). "Graph Drawing by Force-Directed Placement". Software– Practice & Experience (Wiley)http://citeseer.ist.psu.edu/viewdoc/download;?doi=10.1.1.13.8444&rep=rep1&type=pdf
Barnes-Hut on GPU&CPUJ. Barnes and P. Hut. A hierarchical O(N log N) force-calculation algorithm. Nature, 324(4), December 1986http://www.nature.com/nature/journal/v324/n6096/abs/324446a0.html A. S. Bhatele. Review ofBarnes-Hut Implementation in several parallel programming models, May 2006http://charm.cs.uiuc.edu/~bhatele/academics/uiuc/cs498lvk_report_bhatele.pdf
Burtsher on GPUM. Burtscher and K. Pingali. An Efficient CUDA Implementation of the Tree-based Barnes Hut n-Body Algorithm.Chapter 6 in GPU Computing Gems Emerald Edition, pp. 75 - 92. January 2011http://www.gpucomputing.net/?q=node/1314
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 28 / 30
IntroductionOSINT process
NetglubPerspectives
Work in progressGlubbyDemonstration
Demonstration
Step 2/2 : GlubbyUbiGraph (http://ubietylab.net/ubigraph/)Glubby : Fruchterman-Reingold on GPU&CPUGlubby : Barnes-Hut on CPUGlubby : Barnes-Hut + Burtsher on GPU
DisclaimerIt’s still rough around the edges !No user’s features (or just few for testing) for the moment, that’snot the purpose, it’ll be a library and/or a component for Netglub2.0 branch (or Netglub NG) !But it’s awesome... ...or not ,
[email protected] - 2011/05/19 Netglub : Really Open Source Information Gathering
CC A-NC-SA 3.0 License
Page 29 / 30
Netglub : Really Open Source InformationGathering
http://www.netglub.orgThanks for your attention.
Any questions ? (one at a time & slowly please ,)
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.