Tracking large game assets with Git LFS

TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY

Tracking game assets with Git LFS

@kannonboyPhoto: Le Monde en Vidéo

@kannonboy


@kannonboy

@kannonboy@kannonboyPhoto: Le Monde en Vidéo

@kannonboy

Git LFS!

Git LOB!


@kannonboy

!


Git LFS!

TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY

Tracking game assets with Git LFS

@kannonboy

G I T L F S

T H E P R O B L E M W I T H B I G F I L E S

Agenda

C O N V E R T I N G Y O U R R E P O

T I P S F O R T E A M S

@kannonboy

data model

master

fad3d..cat .git/refs/heads/master$

fad3dd41d0cf3d1b6aa2d8ad0549ab2fcb1575d1

@kannonboy

master

98ca9..

bab1e..

fad3d..cat .git/refs/heads/master$

fad3dd41d0cf3d1b6aa2d8ad0549ab2fcb1575d1

@kannonboy“Directed Acyclic Graph”

master

98ca9..

bab1e..

fad3d..

434bb..tree

bab1e..parent

Tim P <kannonboy@…> 1455209277 -0800committer

Tim P <kannonboy@…> 1455209277 -0800author

My life is my commit message.

git cat-file -p 98ca9$

@kannonboy

git cat-file -p 434bb

ace23..100644 blob .gitignoredbdbd..100644 blob .gitattributesa0bc3..040000 tree Assets33d33..040000 tree ProjectSettingsb1de7..100755 blob build.sh7011e..100755 blob README.md

typefilemode SHA-1

master

98ca9..

bab1e..

fad3d..$

434bb..

@kannonboy

master

98ca9..

bab1e..

fad3d..

434bb..

@kannonboy

98ca9..

bab1e..

fad3d..

master

@kannonboy

98ca9..

bab1e..

fad3d..

master

@kannonboy

50mb

98ca9..

bab1e..

fad3d..

master

@kannonboy

50mb100mb98ca9..

bab1e..

fad3d..

master

@kannonboy

50mb100mb150mb

98ca9..

bab1e..

fad3d..

master

@kannonboy

@kannonboy

@kannonboy

@kannonboy

(Large File Storage)

Git LFS

@kannonboy

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

$

LFS store

Git host

@kannonboy

☞

☞

☞

Git host

LFS store

$

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

@kannonboy

LFS store

git push$

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

Git host

@kannonboy

LFS store

git push$

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

Git host

@kannonboy

LFS store

git push$ ☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

Git host

@kannonboy

git pull$

LFS store

Git host

@kannonboy

git pull$

LFS store

Git host

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

4749d..

bdd12..

778aa..

@kannonboy

git pull$

LFS store

Git host

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

4749d..

bdd12..

778aa..

@kannonboy

git checkout bab1e$

LFS store

Git host

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

4749d..

bdd12..

778aa..HEAD

@kannonboy

git checkout bab1e$

LFS store

Git host

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

4749d..

bdd12..

778aa..HEAD

@kannonboy

https://git-lfs.github.com/spec/v1version

sha256:325ddfb…oid

29342295size

git cat-file -p 4749d$ ☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

4749d..

bdd12..

778aa..

@kannonboy

$ brew install git-lfs

$ git lfs install

@kannonboy

$ cat ~/.gitconfig

[filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true

@kannonboy

$ git lfs track “*.mp4”

$ cat .gitattributes

*.mp4 filter=lfs diff=lfs merge=lfs -text

@kannonboy

massive_video.mp4

Work tree

dev

.git/lfs/objects

Clean filter(git-lfs clean)

☞

Index

massive_video.mp4

$

.git/objects

git add

@kannonboy

$

dev

.git/lfs/objects

Smudge filter(git-lfs smudge)

Work tree

massive_video.mp4

☞

Commit tree

massive_video.mp4.git/objects

LFS Store

git checkout

@kannonboy

.git/lfs/objects

.git/objectsHosted repo

LFS store

git push / pull

@kannonboy

$ ls .git/hooks/

commit-msg.sample post-update.sample pre-commit.sample pre-push ...

@kannonboy

$ git push

Git LFS: (12 of 13 files, 1 skipped) 168.75 MB / 180.87 MB, 12.12 skipped

Counting objects: 22, done. ...

@kannonboy

$ git pull

remote: Counting objects: 3, done. ... Downloading massive_video.mp4 (38.79 MB) ... 1 file changed, 2 insertions(+)

@kannonboy

$ git clone ssh://git@bitbucke..

Cloning into ‘big_repo’ ... Downloading massive_video.mp4 (38.79 MB) ... Checking out files: 100% (13/13), done.

@kannonboy

POST .../repo.git

I need object cafebabe

@kannonboy

/info/lfs/objects/batch{“objects”:[

{“oid”: “cafebabe...”,“size”: 40689401}, ...

],“operation”: “download”}

@kannonboy

200 OK

{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {

“download”: { “href”: “https://…/lfs/cafebabe…”,

@kannonboy

cafebabe is over there

} }

...

@kannonboy

200 OK

{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {

“download”: { “href”: “https://…/lfs/cafebabe…”,

@kannonboy

cafebabe is over there

“header”: {“Authorization”: “JWT eyJ0eXA…”,

} } }

...

@kannonboy

$ ssh git@bitbucket git-lfs-authenticate \ project/repo.git download

where is the LFS API?

@kannonboy

@kannonboy


{“href”: “https://…/lfs/objects/batch”,“header”: {“Authorization”: “JWT eyJ0eXA...”

}}


the LFS API is over there

@kannonboy

@kannonboy


{“href”: “https://…/lfs/objects/batch”,“header”: {“Authorization”: “JWT eyJ0eXA...”

}}


the LFS API is over there

@kannonboy

@kannonboy

LFS aware Git server LFS storeDev

git clone https://..

repo data

POST /info/lfs/objects/batch

LFS objects hypermedia

GET …/<objectSHA>

smudge filter

happens once per

file checked out

@kannonboy

LFS storeDev

git lfs clone https://..

repo data

GET …/<objectSHA>

batched

smudge filter

subtle difference!

POST /info/lfs/objects/batch

LFS objects hypermedia

LFS aware Git server

@kannonboy

# git pull with LFS disabled $ git -c filter.lfs.smudge= \ -c filter.lfs.required=false pull

# fetch LFS objects as batch $ git lfs pull

Speeding up pulls

@kannonboy

# define a git alias $ git config --global alias.plfs "\!git \ -c filter.lfs.smudge= \ -c filter.lfs.required=false pull \ && git lfs pull"

# then simply... $ git plfs

Speeding up pulls

@kannonboy

Converting to Git LFS

☞

@kannonboy

Converting to Git LFS

☞

98ca9..

bab1e..

fad3d..

100mb150mb


98ca9..

bab1e..

fad3d.. ☞

100mb150mb150mb!?!?


@kannonboy

git filter-branch

$ git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch big_video.mp4’ \ --prune-empty --tag-name-filter cat -- --all

DON’T DO

THIS!

@kannonboy

$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all

@kannonboy

DON’T DO

THIS

EITHER!

@kannonboy

BFG Repo-Cleaner

@kannonboy

by @rtyley

@kannonboy

BFG Repo-Cleaner

@kannonboy

10-720x faster than filter-branch

built to kill history

Git LFS support

by @rtyley

@kannonboy

$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all

@kannonboy

DON’T DO

THIS

EITHER!

@kannonboy

$ brew install bfg

$ bfg —-convert-to-git-lfs “*.mp4” --no-blob-protection

@kannonboy

What to track?Yes Maybe No

SFXMusic

TexturesSpritesheets

FMV

Code

FontsMaterials

Text

MeshesAnimations

Scenes

if they’re big or change frequently

big stuff

@kannonboy

$

317352 angrybots194424 angrybots/Assets112884 angrybots/Assets/Textures32220 angrybots/Assets/Sounds24460 angrybots/Assets/Objects…

du -k angrybots | sort -nr

What to track?

@kannonboy

Repofactoring

@kannonboy

@kannonboy

Identifying large objects

github.com/bloomberg/repofactor

by @hashpling

@kannonboy

$

a295ef4… 102437 953722cc7063… 152171 140443

blob SHA size on disk average blob size

generate-larger-than 50000


@kannonboy

, PNG, 512 x 512, WAVE audio, 16 bit

a295ef4… 102437 95372 2cc7063… 152171 140443

$ generate-larger-than 50000 \| add-file-info


@kannonboy

…/ReBirth.png a295ef4… 102437 95372, PNG… …/bot_die.wav 2cc7063… 152171 140443, WAVE…

$ generate-larger-than 50000 \ | add-file-info

$ report-on-large-objects big-stuff.txt

paths

> big-stuff.txt


@kannonboy

…/bot_die.wav 2cc7063… 152171 140443, WAVE… …/ReBirth.png a295ef4… 102437 95372, PNG…

$ generate-larger-than 50000 \ | add-file-info

$ report-on-large-objects big-stuff.txt

> big-stuff.txt


\| sort -k3nr

order by average blob size

@kannonboy

$ bfg —-convert-to-git-lfs “*.wav” --no-blob-protection


$ bfg —-convert-to-git-lfs “*.png”--no-blob-protection

run once per pattern

@kannonboy

Enable in Bitbucket

@kannonboy

@kannonboy

@kannonboy

Tips for teams

@kannonboyBeware merge conflicts @kannonboy

@kannonboy

…meanwhile in

@kannonboy

Locking

@kannonboy

Source: etonline.com/music

@kannonboy

master

feature0

feature1

File locking (single branch model)

LAME

@kannonboy

master

feature0

feature1

File locking (multi branch model)

@kannonboy

master

feature0

feature1


@kannonboy

master

feature0

feature1


@kannonboy

master

feature0

feature1


@kannonboy

$ git lfs lock <path>

$ git lfs unlock <path>

$ git lfs locks [-i id] [-p path]

@kannonboy

Until then…

@kannonboyTeamwork @kannonboyTeamwork

@kannonboy

$ git lfs fetch

master

feature0

feature1

@kannonboy

$ git lfs fetch

master

feature0

feature1

--recent

@kannonboy

$ git lfs fetch

master

feature0

feature1

--recent

T-minus 7 days

@kannonboy

$ git config lfs.fetchrecentalways “true”

lfs.fetchrecentrefsdays

lfs.fetchrecentremoterefs

lfs.fetchrecentcommitsdays

(default = 7)

(default = 0)

$ git lfs fetch --recent

(default = true)

@kannonboy

$

master

feature0

feature1

git lfs prune

@kannonboy

$

master

feature0

feature1

--recent(7 days)

prune(10 days)

git lfs prune

@kannonboy

lfs.pruneoffsetdays

lfs.pruneverifyremotealways

(default = 3)

Set this!

$ git lfs prune

@kannonboy

Server-side deletion

@kannonboyFetch the bare necessitiesFetch the bare necessities @kannonboy© Disney

@kannonboy

# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/Textures/**

# for an audio engineer $ git lfs fetch --include Assets/Sounds/**

$ git config lfs.fetchexclude Assets/Textures/**

$ git config lfs.fetchinclude Assets/Sounds/**

@kannonboyIDEs & GUI tools @kannonboy

@kannonboy

@kannonboy

@kannonboy

SourceTree

@kannonboy

version 4.3+Bitbucket Server

go.atlassian.com/git-lfsdocs

github.com/github/git-lfssource

available now!bitbucket.org

Lookingfor

more?

For occasional Git, JIRA& Bitbucket trivia

@kannonboy

unlimited private and public repositories for teams of up to 5 Git LFS storage tier builds and deployments tier with Bitbucket Pipelines pull requests, branch controls, JIRA integration, etc

go.atlassian.com/gaming

FREE

@kannonboy

Tweet your game or app with the hashtag

#BuiltwithBitbucket

for a chance to be featured on the Bitbucket gaming

page

Tracking large game assets with Git LFS

Software