Global Software Development Kevin Loken Group Technical Director, EA
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Global Software Development
Global Software DevelopmentKevin LokenGroup Technical Director, EA
1
Electronic Arts20+ Development studios worldwide#1 publisher on PlayStation 4 and Xbox One8,400+employeeshttp://ir.ea.com
#
Electronic Arts & Perforce5,001EALicenses15+YearsUsage10,000+check-insIn a day160 Perforce Servers 350+TBData180ProxyServers
#
EA has been using Perforce for over 15 years now. The first server was setup for our Tools & Libraries group with 5 users.Then we added it to FIFA, with split development between UK and EA Vancouver, Proxy server helping out significantlyWe currently have over 160 perforce servers setup, 180 proxy servers, and about half a dozen edge serversTotal data under storage is over 350 TB, growing every dayOur high traffic servers have meta-data databases that are approaching 1 TB (800 MB)
3
Global Software Development
Global Software DevelopmentIts not 2 guys in a garage anymoreHundreds of staffArt & AnimationAudioCodeOnline services & infrastructureQA, Certification, ManufacturingNot in one geographic location anymore
#
Global Software DevelopmentFun is proportional to productivityProductivity is about making the remote experience the same as the local experiencePerforce offers a number of features that help:Proxy serversEdge serversDVCS
#
Global Software DevelopmentPerforce Proxy serverEasy to setupEssentially zero maintenanceCull old files on a regular basisExtremely robustGreat if all you do is sync, edit & submitTerrible if you diff, revert, switch streams
#
Global Software DevelopmentPerforce Edge serverComplicated to setup too complicated to show a movie Needs backup, checkpointsThere is data that is specific and unique to the edge serverp4 submit is longer than localAll other operations just like local
#
Global Software DevelopmentStill limited by TCP protocol and network latencyBut I want it to go faster!Enable parallel syncWAN acceleration
#
WAN acceleration SilverPeak, Riverbed, SonicWALL, MACH5
9
Global Software DevelopmentArt contentEngine developmentGame developmentOnline servicesSensitive dataLocalizationBinary data storesTools & CIProxy and Edge servers are greatAssumes a single Perforce Helix serverLots of Perforce Helix servers for a game:
#
Global Software DevelopmentRisk of moving to a single repository is too highHuman error is still a factor in our workBad triggersBad upgradesTime to execute commands increasesCheckpointsRestorationFinding a window to upgrade would be impossibleWorld-wide operations, games on different cycles, never a good time
#
Global Software DevelopmentIt would be nice to share file history between serversEspecially between game engine and game teamOver the years we have developed a ton of scripts to do thisBut there is something new
#
Global Software DevelopmentPerforce Helix DVCS TechnologyAll documentation points to individual usageAlso works server-server!This allows us to preserve history between serversIt simplifies integration work flows for harvesting changesCustom scripts are gone, replaced by DVCS features
#
Perforce Helix DVCS
#
The basic flow of code begins from the common ancestry point on frostbite.Having establish a common history, we `p4 fetch` to a branch on the game perforce server.This new branch (//mule/integ/) is integrated into the dev line (//mule/dev/...).We finish the box by executing `p4 fetch` on frostbite (or a `p4 push` mule perforce server)And integrate into the origin common ancestor (//fmule/integ/...)
The red boxes indicate that these branches should be consider read-only on their server. Only the user/service account that does the fetch should have write access to those branches.
14
Perforce Helix DVCS
#
15
Perforce Helix DVCS
#
16
Perforce Helix DVCS
#
17
Perforce Helix DVCS
#
18
Perforce Helix DVCS
#
19
Perforce Helix DVCS
#
20
Perforce Helix DVCSEnables sharing of code across serversFirst-class feature in Perforce no more custom scriptsWorks through proxy and edge serversA way of mitigating risk if a huge mono-repo is not for you
#
Security
EA is a target
#
EA is a target
#
SecurityEA Security modelOld days outside the network - bad, inside the network - safeToday zero trust model
#
SecurityPerforce security modelsProtect tableUser managementAD SecurityStarting to directly embed AD security groups into Perforce groupsSimplifies management one source of truthPotential downside: Not everyone is aware that adding a user to this group, means they get Perforce access
#
SecurityPerforce Helix Threat DetectionRelies on audit logsTake a lot of space!Intersects with our Employee Monitoring & Privacy PolicyHere come the lawyers
#
Perforce Helix Threat DetectionUnderstand privacy issues of your jurisdictionSwitzerland: No, cant monitor employees.Germany: Need approval of work councilFrance: Need approval of individual employeesCanada: Need to notify employeesUS: Dont need to notify employees, but do anywayContractors: Notify contracting agency in addition to country specific rules
#
Perforce Helix Threat DetectionHandling the privacy issuesBuild a system that intercepts and cleans all audit logsIntegrated with HR systems Custom rule sets that allow filtering based on local rulesOnly pass the cleaned audit logs on to Threat Detection
#
Perforce Helix Threat DetectionProof of Concept3 months of audit logs from 2 serversPost processingIdentified 10 unusual or suspicious usersIndividually handle each case, most of them explainableDue to post-process nature some instances couldnt be fully investigatedResults made the investment in Threat Detection a no-brainer for us
#
Funny story, SRM gets tagged as moochers because they only sync code
30
Perforce Helix Threat DetectionGoing liveNear real-time processing of audit logs from all serversIntrusion detection vs. Incident responsePresent case to legal department to get investigation authorizationInvestigate once authorization is achievedFutureIntegrate with other data sources like QRadar
#
Perforce Helix Threat DetectionResultsProof of Concept results made investment in Threat Detection a no-brainer for EAWide ranging legal issues involved, especially for multi-national companiesBecomes exponentially more powerful as you integrate with other data sources
#
Thank you!Contact:Kevin [email protected]
Related Documents
![Scripting with Perforce · Scripting with Perforce Using the Perl and Ruby interfaces (c) ... fs = p4.run_fstat( “file.c” ) head = fs[ “headRev” ] (c) 2003 Perforce Software,](https://static.cupdf.com/doc/110x72/5b1e82027f8b9af1328b8e54/scripting-with-perforce-scripting-with-perforce-using-the-perl-and-ruby-interfaces.jpg)
