Transcript
My trip to the
BACK SIDE
@LemaNahuel
I press keys on javascript
That is FWTV?
FWTV is the first Latin WebTV channel with designed exclusively for the Internet, where viewers can interact with programs offered both live and on-demand, anytime, anywhere content.
https://www.fwtv.tv/
@Obaca2015 Me
A long time ago ... (2013-2014)
fwtv.tv
fwtv.tv/api/*
fwtv.tv/admin/*
Heroku with 4 dynos
● Monolithic project● +150 endpoints and growing● callback hell● Frozen dependencies (Node .08, Mongoose,
Redis, Express, etc)
● No Documentation or Tests● Filesystem is no more scalable● Very long Files (average from 1000 to 2500 lines)● ...
... ART* 500 ms - 1000 ms or more … ¬¬
*Average Response Time
Preparing the ground for v2
Se we start with the obvious steps
api.
.tv
admin.
Like miners we start to separate the projects
● Tests implementation● Added documentación● Extreme implementation of EsLint and jsHint● Updated dependencies and Node to 0.12● Restructured the entire filesystem
○ routes/(public || private)/*.js
○ controllers/(public || private)/*.js
● Replaced all long files with small modules● ...
… ART meeh … ¬¬
… but now scale better!
v2.0 > v2.1 - Go functional!
Standardize the use of lodash● aprox 100 implementations of _.each()● more methods implemented as _.pluck(), _.
indexOf(), _.find(), _.without(), _.omit(), etc.
_.each(req.body.genres, (genre) => {
if (_.isObject(genre)) {
newGenres.push(genre._id);
} else {
newGenres.push(genre);
}
});
var genres = req.body.genres,
genre = null,
i = 0,
l = genres && genres.length ? genres.length : 0;
for (i; i < l; i++) {
genre = genres[i];
if (typeof genre === 'object') {
newGenres.push(genre._id);
} else {
newGenres.push(genre);
}
}
For example:_.each()
… ART with no improvements …
… but the code is more readable!
v2.1 > v2.2 - only what you need
Implementation of “Extender”● Now you can specify which fields are required and
which are not.● Used especially for translations and data uses only
the API or the admin.
For example:
function getTranslations() { if (LOCALE.indexOf('es') === -1) { return ',translations(' +LOCALE + '(description))'; }
return '';}
var fields = '?fields='; fields += 'landing'; fields += ',popularShows(_id,logo,slug,title,description,slogan' + getTranslations() + ')';
return $http.get(PATHS.HOME + fields);
Query strings added to Page and Limit● Skip● Limit● LastId
For example:
return $http.get(PATHS.SHOWS, {
params: {
limit: params.limit || 12,
lastId: params.lastId,
skip: params.skip
}
});
… ART barely noticeable …
… but the weight is lowered 100 kb - 250 kb to 1b-30 kb!
v2.2 > v2.3 - the real meaning of parallel
ASYNC● _.each() >
○ async.each()● callback hell >
○ async.waterfall() ○ async.parallel()
For example:async.waterfall()
getHome(function(home) { setPopularAndLiveShows(home, function(popularShows, liveNow, newNow){ home.popularShows = popularShows; home.liveNow = liveNow; home.newNow = newNow;
setFeaturesVideos(home, function(featuredVideos){ home.featuredVideos = featuredVideos; helpers.handleResponse(res, null, home); }); });});
async.waterfall([function(cb) { getHome(function(home) { cb(null, home); });}, function(cb) { setPopularAndLiveShows(home, function(popularShows, liveNow, newNow) { home.popularShows = popularShows; home.liveNow = liveNow; home.newNow = newNow; cb(null, home); });}, function(home, cb) { setFeaturesVideos(home, function(featuredVideos){ home.featuredVideos = featuredVideos; cb(null, home); });}], function(err, home) { helpers.handleResponse(res, null, home);});
For example:async.parallel()
async.parallel([function(cb) { setPopularVideos(home, likedProgramIds, function(popularVideos) { home.popularVideos = popularVideos; cb(null); }); }, function(cb) { setGenres(home, function(genres) { if(genres) { home.genres = genres; } cb(null, home); });}], function(err) { cb(null, home);});
For example:async.each()
async.each(shows, function(show, cb) { ScheduledPromo.count({
program: show._id }).exec(function(err, scheduleds) { Program.findByIdAndUpdate(show._id, { $set: { 'hasElements.scheduled': !!scheduleds } }).exec(function(err) { cb(err); }); }); }, function(err) { helpers.handleResponse(res, null, { success: !err });});
… ART incredible striking …
… it is now 500 ms - 1000 ms to 150 ms - 300 ms!
v2.2 > v3 - Sh*t just got real
Upgrade to Node 4.* and npm dependencies
… ART still the same …
… but consumption of RAM of Heroku decreased significantly!
v3 > v3.1 - sugar Marty ... syntaxis sugar everywhere
Upgrade to Node 5.* and npm dependencies● Minimum implementation of ES6
○ EsLint + ES6
○ const y let
○ arrow functions
○ classes
… ART unimproved and falling consumption of RAM ...
… but the code is much more readable
The journey so far● All up-to-date (Node && NPM)● Functional programming by lodash● Asynchronism, parallelism and callback hell resolved with
async
● ES6 minimally implemented● ART between 100 ms - 300 ms
still something missing …
… review all the access to MongoDB
v3.1 > v3.2 - the final fight!
The scenario:● Made with Mongoose● ~400 queries● most carried out by the front (.tv) and the
apps
I’m Mr. Meeseeks!! Look at me!
Step 1:● +400 queries 1 by 1:
○ .find() >
■ .findById() || .findOne()
○ .update() >
■ .findByIdAndUpdate() || .findOneAndUpdate()
○ .remove() >
■ .findByIdAndRemove() || .findOneAndRemove()
For example:.find() > .findById()
Model.find({ _id: req.params.id }).populate('video grouping program genres').exec((err, doc) => { helpers.handleResponse(res, err, doc);});
Model.update({ _id: req.params.id }, {}).exec((err, doc) => { helpers.handleResponse(res, err, doc);});
Model.remove({ _id: req.params.id }).exec((err, doc) => { helpers.handleResponse(res, err, doc);});
Model.findById(req.params.id) .populate('video grouping program genres') .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndUpdate(req.params.id, {}, { new: true }) .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndRemove(req.params.id) .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Step 2:● Implemented .lean() method on all .
find*() before .exec()
For example:.lean()
Model.findById(req.params.id) .populate('video grouping program genres') .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndUpdate(req.params.id, {}, { new: true }) .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndRemove(req.params.id) .exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findById(req.params.id) .populate('video grouping program genres') .lean().exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndUpdate(req.params.id, {}, { new: true }) .lean().exec((err, doc) => { helpers.handleResponse(res, err, doc); });
Model.findByIdAndRemove(req.params.id) .lean().exec((err, doc) => { helpers.handleResponse(res, err, doc); });
ok, and now ...
… the ART down to 5 ms - 35 ms!! MOTHERF*CKAAA!!!!
Sum-up
● Functional, clean and less code● Lightweight requests● Low consumption of RAM● Efficient queries● Same amount of dynos, greater capacity● Weekly NCU
Questions?
How many Mr. Meeseeks did you find?- 9- 13- 15
Thanks!
@LemaNahuel
top related