Lesson: Web Programming(3)ce.sharif.edu/courses/96-97/1/ce419-1/resources/root/wp-3.pdfGolang, NodeJs, MongoDB, PostgreSQL, Redis Docker, Git, NPM, YUIDoc, ... To tell Git to start

Lesson: Web Programming(3)Omid Jafarinezhad

Sharif University of Technology

Materials

● HTTP, JavaScript, CSS, HTML5, ReactJs, Flow, Progressive Web App

● Golang, NodeJs, MongoDB, PostgreSQL, Redis

● Docker, Git, NPM, YUIDoc, Jest, WebPack, Gulp, Browserify, Locust

● (Optional/Research) Kubernetes, InfluxDB, RabbitMQ, gRPC, Ansible

Git

Helps software developers to work together and maintain a complete history of their work

GitA version control system (VCS) allows you to track the history of a collection of files

Git is a distributed revision control and source code management system with an emphasis on speed

> sudo apt-get install git

> sudo yum install git

> git --version

Git - Life CycleA Git repository contains the history of a collection of files starting from a certain directory

Amend: After committing, if you realize something is wrong, then you correct the last commit and push the changes to the repository

Working treeThe working tree contains the set of working files for the repository

You can modify the content and commit the changes as new commits to the repository

CommitWhen you commit your changes into a repository this creates a new commit object in the Git repository. This commit object uniquely identifies a new revision of the content of the repository

This revision can be retrieved later, for example, if you want to see the source code of an older version

Each commit object contains the author and the committer. This makes it possible to identify who did the change. The author and committer might be different people. The author did the change and the committer applied the change to the Git repository. This is common for contributions to open source projects

Staging areaThe staging area is the place to store changes in the working tree before the commit

The staging area contains a snapshot of the changes in the working tree (changed or new files) relevant to create the next commit and stores their mode (file type, executable bit)

The staging area (aka index) is a container where git collects all changes which will be part of the next commit. If you are editing a versioned file on your local machine, git recognizes that your file is modified - but it will not be automatically part of your next commit and is therefore unstaged. Staging the file will put the file into the staging area (index). The next git commit will transfer all items from staging area into your repository

RevisionRepresents a version of the source code

Git implements revisions as commit objects (or short commits ). These are identified by an SHA-1 hash

TagA tag points to a commit which uniquely identifies a version of the Git repository. With a tag, you can have a named point to which you can always revert to. You can revert to any point in a Git repository, but tags make it easier. The benefit of tags is to mark the repository for a specific reason, e.g., with a release.

Branches and tags are named pointers, the difference is that branches move when a new commit is created while tags always point to the same commit. Tags can have a timestamp and a message associated with them

BranchGit supports branching which means that you can work on different versions of your collection of files, A branch allows the user to switch between these versions so that he can work on different changes independently from each other. One of the branches is the default (typically named _master ). The default branch is the one for which a local branch is automatically created when cloning the repository

● For example, if you want to develop a new feature, you can create a branch and make the changes in this branch. This does not affect the state of your files in other branches. For example, you can work independently on a branch called production for bugfixes and on another branch called feature_123 for implementing a new feature

Brach (2)A branch is a named pointer to a commit. Selecting a branch in Git terminology is called to checkout a branch. If you are working in a certain branch, the creation of a new commit advances this pointer to the newly created commit ([p:231]... branches move when a new commit is created ...)

Each commit knows their parents (predecessors). Successors are retrieved by traversing the commit graph starting from branches or other refs, symbolic references (for example: HEAD) or explicit commit objects. This way a branch defines its own line of descendants in the overall version graph formed by all commits in the repository

Exampleterminal prompt

A working directory

To initialize a Git repository here

to see what the current state of our project

I created a file called octocat.txt in the octobox repository

To tell Git to start tracking changes made to octocat.txt, we first need to add it to the staging area by using git add

To store our staged changes we run the commit command with a message describing what we've changed

You also can use wildcards if you want to add many files of the same type

I put some(.txt file) in a directory named "octofamily" and some others ended up in the root of our "octobox" directory

So we've made a few commits. Now let's browse them to see what we changed. Think of Git's log as a journal that remembers all the changes we've committed so far, in the order we committed them

To push our local repo to the GitHub server we'll need to add a remote repository

The push command tells Git where to put our commits

> Push our local changes to our origin repo

Let's pretend some time has passed. We've invited other people to our GitHub project who have pulled your changes, made their own commits, and pushed them. We can check for changes on our GitHub repository and pull down any new changes

Fast-Forward

Compare HEAD by Pulled files :-)

1,4 range (line)- removed+ added

You can unstage files by using the git reset command

create a branch called clean_up You can switch branches using the git checkout <branch> command

Switching Back to master

We're already on the master branch, we can merge our changes from the clean_up branch into the master branch

Basic Branching and Merging ExampleLet’s go through a simple example of branching and merging with a workflow that you might use in the real world. You’ll follow these steps:● Do some work on a website● Create a branch for a new story you’re working on● Do some work in that branch

At this stage, you’ll receive a call that another issue is critical and you need a hotfix. You’ll do the following:● Switch to your production branch● Create a branch to add the hotfix● After it’s tested, merge the hotfix branch, and push to production● Switch back to your original story and continue working

You’ve decided that you’re going to work on issue #53 in whatever issue-tracking system your company uses

Now you get the call that there is an issue with the website, and you need to fix it immediately. With Git, you don’t have to deploy your fix along with the iss53 changes you’ve made, and you don’t have to put a lot of effort into reverting those changes before you can work on applying your fix to what is in production

You’ll notice the phrase “fast-forward” in that merge. Because the commit C4 pointed to by the branch hotfix you merged in was directly ahead of the commit C2 you’re on, Git simply moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit’s history, Git simplifies things by moving the pointer forward because there is no divergent work to merge together – this is called a “fast-forward.”

After your super-important fix is deployed, you’re ready to switch back to the work you were doing before you were interrupted

Suppose you’ve decided that your issue #53 work is complete and ready to be merged into your master branch. In order to do that, you’ll merge your iss53 branch into master, much like you merged your hotfix branch earlier

This looks a bit different than the hotfix merge you did earlier. In this case, your development history has diverged from some older point. Because the commit on the branch you’re on isn’t a direct ancestor of the branch you’re merging in, Git has to do some work. In this case, Git does a simple three-way merge, using the two snapshots pointed to by the branch tips and the common ancestor of the two.Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parentIt’s worth pointing out that Git determines the best common ancestor to use for its merge base

Now that your work is merged in, you have no further need for the iss53 branch. You can close the ticket in your ticket-tracking system, and delete the branch

Occasionally, this process doesn’t go smoothly. If you changed the same part of the same file differently in the two branches you’re merging together, Git won’t be able to merge them cleanly. If your fix for issue #53 modified the same part of a file as the hotfix branch, you’ll get a merge conflict that looks something like this

Your file contains a section that looks something like this

This means the version in HEAD (your master branch, because that was what you had checked out when you ran your merge command) is the top part of that block (everything above the =======), while the version in your iss53 branch looks like everything in the bottom part. In order to resolve the conflict, you have to either choose one side or the other or merge the contents yourself

Cherry-pickProblem is found on the Production branch. A fix for the problem is developed in commit H, but commit G does not need to be applied to the Development branch

Commit H is cherry-picked onto the Development branch, resulting in commit H'. Note that the changes made in commit G are not included on the Development branch

ResetOn the commit-level, resetting is a way to move the tip of a branch to a different commit

This can be used to remove commits from the current branch (a simple way to undo changes)

> git reset HEAD~~HEAD is now at 326c9f commit message

> git reset HEAD~2HEAD is now at 326c9f commit message

RevertReverting undoes a commit by creating a new commit. This is a safe way to undo changes, as it has no chance of re-writing the commit history

> git revert HEAD~2

RebaseRebasing is the process of moving or combining a sequence of commits to a new base commit. Rebasing is most useful and easily visualized in the context of a feature branching workflow

The primary reason for rebasing is to maintain a linear project history

● For example, consider a situation where the master branch has progressed since you started working on a feature branch. You want to get the latest updates to the master branch in your feature branch, but you want to keep your branch's history clean so it appears as if you've been working off the latest master branch. This gives the later benefit of a clean merge of your feature branch back into the master branch.

Docker

a helpful tool for packing, shipping, and running applications within “containers”

What are “containers” and “VMs”?Containers and VMs are similar in their goals:

● to isolate an application and its dependencies into a self-contained unit that can run anywhere

● remove the need for physical hardware, allowing for more efficient use of computing resources, both in terms of energy consumption and cost effectiveness

The main difference between containers and VMs is in their architectural approach

● Docker share the host system’s kernel with other containers

Virtual machines run guest operating systems—note the OS layer in each box. This is resource intensive, and the resulting disk image and application state is an entanglement of OS settings, system-installed dependencies, OS security patches, and other easy-to-lose, hard-to-replicate ephemera

Containers can share a single kernel, and the only information that needs to be in a container image is the executable(App A, B, C) and its package dependencies (Bin/Lib), which never need to be installed on the host system. Because they contain all their dependencies, there is no configuration entanglement; a containerized app “runs anywhere”

guest machine host machine

VM

● New● Setting and Install● Start

Docker

● Pull● Run (Stop/start)

Where does Docker come in?Docker is an open-source project based on Linux containers. It uses Linux Kernel features like namespaces and control groups to create containers on top of an operating system

Containers are far from new; Google has been using their own container technology for years. Others Linux container technologies include Solaris Zones, BSD jails, and LXC, which have been around for many years

So why is Docker all of a sudden gaining steam?Ease of use: Docker has made it much easier for anyone — developers, systems admins, architects and others — to take advantage of containers in order to quickly build and test portable applications

Speed: Docker containers are very lightweight and fast

Docker Hub: Docker users also benefit from the increasingly rich ecosystem of Docker Hub, which you can think of as an “app store for Docker images”

Modularity and Scalability: Docker makes it easy to break out your application’s functionality into individual containers

Install Docker (CE/EE)Docker is available in two editions: Community Edition (CE) and Enterprise Edition (EE)

> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - > sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"> sudo apt-get update> sudo apt-get install -y docker-ce> sudo systemctl status dockerdocker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2016-05-01 06:53:52 CDT; 1 weeks 3 days ago Docs: https://docs.docker.com Main PID: 749 (docker)

-- Ubuntu 16.04

Docker Command Without Sudo (Optional, Dev)By default, running the docker command requires root privileges — that is, you have to prefix the command with sudo. It can also be run by a user in the docker group, which is automatically created during the installation of Docker. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you'll get an output like this

If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group

docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.See 'docker run --help'.

> sudo usermod -aG docker ${USER}> su - ${USER}

1. Add your username to the docker group

2. apply the new group membership(without logout)

A brief explanation of containersAn image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files

A container is a runtime instance of an image—what the image becomes in memory when actually executed. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so● Containers run apps natively on the host machine’s kernel. Containers can

get native access, each one running in a discrete process, taking no more memory than any other executable

Docker image repositories

https://hub.docker.com

https://store.docker.com

Pull an image from Docker Hub> docker pull debian

Using default tag: latestlatest: Pulling from library/debianfdd5d7827f33: Pull completea3ed95caeb02: Pull completeDigest: sha256:e7d38b3517548a1c71e41bffe9c8ae6d6d29546ce46bf62159837aad072c90aaStatus: Downloaded newer image for debian:latest

Pull an image from Docker Hub (2)> docker pull ubuntu:14.04

14.04: Pulling from library/ubuntu5a132a7e7af1: Pull completefd2731e4c50c: Pull complete28a2f68d1120: Pull completea3ed95caeb02: Pull completeDigest: sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2Status: Downloaded newer image for ubuntu:14.04

Displaying Docker ImagesTo see the list of Docker images on the system, you can issue the following command

> docker imagesubuntu-java latest 389560217a19 3 weeks ago 889MBubuntu latest 8b72bba4485f 3 weeks ago 120MBconsul latest c6f7042bd0f8 2 months ago 51.7MBmongo latest b39de1d79a53 2 months ago 359MBinfluxdb latest d913baeaf36b 2 months ago 227MBpostgres latest 33b13ed6b80a 2 months ago 269MBnode latest 045ea30913bb 2 months ago 667MBrabbitmq latest bad16bdb4e74 2 months ago 124MBnginx latest b8efb18f159b 2 months ago 107MB

Images and layersA Docker image is built up from a series of layers

Each layer represents an instruction in the image’s Dockerfile

When you use docker pull to pull down an image from a repository, each layer is pulled down separately, and stored in Docker’s local storage area, which is usually /var/lib/docker/ on Linux hosts. You can see these layers being pulled in this example:

> docker pull debian

Using default tag: latestlatest: Pulling from library/debianfdd5d7827f33: Pull completea3ed95caeb02: Pull completeDigest: sha256:e7d38b3517548a1c71e41bffe9c8ae6d6d29546ce46bf62159837aad072c90aaStatus: Downloaded newer image for debian:latest

Docker runRunning of containers is managed with the Docker run command

Assign name and allocate pseudo-TTY (–name, -it)--interactive , -i Keep STDIN open even if not attached--name Assign a name to the container--tty , -t Allocate a pseudo-TTY (pseudoterminal)

> docker run --name test -it debianroot@d6c0fe130dba:/# exit

> docker ps -a | grep testd6c0fe130dba debian:7 "/bin/bash" 26 seconds ago Exited (13) 17 seconds ago test

This example runs a container named test using the debian:latest image. The -it instructs Docker to allocate a pseudo-TTY connected to the container’s stdin; creating an interactive bash shell in the container

Listing of ContainersOne can list all of the containers on the machine via the docker ps command. This command is used to return the currently running containers

> docker ps -aThis command is used to list all of the containers on the system

Run in background and remove if exits--detach , -d Run container in background and print container ID

--rm Automatically remove the container when it exits

>docker run --name ubuntu_bash --rm -i -t ubuntu bashroot@4f47613a44e3:/# exitexit> >docker run --name ubuntu_bash -d --rm -i -t ubuntu bash327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75>

Publish or expose port (-p, –expose)--publish , -p Publish a container’s port(s) to the host

This binds port 8080 of the container to port 80 on 127.0.0.1 of the host machine

$ docker run -p 127.0.0.1:80:8080 ubuntu bash

Set environment variables (-e, –env, –env-file)--env , -e Set environment variables

--env-file Read in a file of environment variables

Use the -e, --env, and --env-file flags to set simple (non-array) environment variables in the container you’re running, or overwrite variables that are defined in the Dockerfile of the image you’re running.

$ docker run --env MYVAR2=foo --env-file ./env.list ubuntu bash

Set metadata on container (-l, –label, –label-file)--label , -l Set meta data on a container

--label-file Read in a line delimited file of labels

A label is a key=value pair that applies metadata to a container

The my-label key doesn’t specify a value so the label defaults to an empty string(""). To add multiple labels, repeat the label flag (-l or --label)

$ docker run -l my-label --label com.example.foo=bar ubuntu bash

Set ulimits in container (–ulimit)$ docker run --ulimit nofile=1024:1024 debian sh -c "ulimit -n"1024

$ docker run -ti node /bin/bashuser@4d04d06d5022:/# ulimit -a...max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimitedopen files (-n) 32000pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200cpu time (seconds, -t) unlimitedmax user processes (-u) 58729….

<type>=<soft limit>[:<hard limit>]

A hard limit is the maximum allowed to a user/process;Increasing hard limit can be done only by root

A soft limit is the effective value right now for that user/process; a soft limit may be increased up to the value of the hard limit

Mount volume--volume , -v Bind mount a volume

$ docker run -v /data:/path-in-container/data -i -t ubuntu bash

Memory Constraints

$ docker run -it -m 300M ubuntu:16.04 /bin/bash

CPU ConstraintsYou can limit CPU, either using a percentage of all CPUs, or by using specific cores.

● The setting is a bit strange -- 1024 means 100% of the CPU, so if you want the container to take 50% of all CPU cores, you should specify 512

● You can also only use some CPU cores using cpuset-cpus

$ docker run -ti --c 512 ubuntu:16.04 /bin/bash$ docker run -ti --cpuset-cpus=0,4,6 ubuntu:16.04 /bin/bash

Listing All Docker Networks> docker network ls

This command can be used to list all the networks associated with Docker on the host

> docker network lsNETWORK ID NAME DRIVER SCOPE2ededf290eeb bridge bridge localc8fec94ed780 docker-net bridge local58a207b3a0d2 host host local365568e1e2bf my-mongo-cluster bridge local

> sudo docker network inspect bridge>sudo docker run –it ubuntu:latest /bin/bash

> sudo docker network inspect bridge

Creating Your Own New Network> docker network create --driver bridge new_nw d0f4e81ce4d629e904f3780840c2c097ba669c8d9abb4e0f18ab700f59683429> sudo docker run -it --network=new_nw ubuntu:latest /bin/bash> sudo docker network inspect new_nw [ { "Name": "new_nw", "Id": "d0f4e81ce4d629e904f3780840c2…….", …... "Config": [ { "Subnet": "172.19.0.0/16", "Gateway": "172.19.0.1" }

Connect a container to a network (–network)--network Connect a container to a network

You can also choose the IP addresses for the container with --ip and --ip6 flags when you start the container on a user-defined network

If you do not specify a different network, new containers are automatically connected to the default bridge network

$ docker run -itd --network=my-net busybox$ docker run -itd --network=my-net --ip=10.10.9.75 busybox

docker inspectReturn low-level information on Docker objects

$ docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES77c6e841638e mongo "docker-entrypoint..." 10 days ago Exited (0) 10 days ago mongo1

$ docker inspect mongo1

Useful Docker Commandsdocker start starts a container so it is runningdocker stop stops a running containerdocker restart stops and starts a containerdocker kill sends a SIGKILL to a running containerdocker rm remove one or more containersdocker rmi remove one or more imagesdocker exec run a command in a running containerdocker save save one or more images to a tar archive (streamed to STDOUT by default)

docker commit create a new image from a container’s changes

$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES77c6e841638e mongo "docker-entrypoint..." 10 days ago Exited (0) 10 days ago mongo-docker

$ docker start 77c6e841638e$ docker start 77 $ docker start 77c $ docker start 77c6$ docker start mongo-docker

> docker run --name ubuntu_bash --rm -i -t ubuntu bashroot@4f47613a44e3:/# exitexit> > docker run --name ubuntu_bash -d --rm -i -t ubuntu bash327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75> docker exec -it ubuntu_bash bashroot@327f7e6b08d6:/# exitexit> docker exec -it ubuntu_bash lsbin dev home lib64 mnt proc run srv tmp varboot etc lib media opt root sbin sys usr> > docker save busybox > busybox.tar

Container OrchestrationDocker Swarm is a native Docker container orchestrator that allows users to treat a group of Docker hosts as a single Docker Engine

Kubernetes is an open-source platform for container deployment automation, scaling, and operations across clusters of hosts. The production ready orchestrator draws on Google’s extensive experience of years of working with Linux containers

Node Package Manager (NPM)

Install Node.js, npm, and stay up-to-date

What is npm?npm makes it easy for JavaScript developers to share and reuse code, and makes it easy to update the code that you’re sharing, so you can build amazing things

npm is installed with Node.js

-- ubuntu > curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -> sudo apt-get install -y nodejs

> curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -> sudo apt-get install -y nodejs

Installing NPM ...

root@73eee145286b:/# which node/usr/local/bin/noderoot@73eee145286b:/# node --versionv8.2.1root@73eee145286b:/# which npm/usr/local/bin/npmroot@73eee145286b:/# npm --version5.3.0

Other Nifty Commands$ npm initcreates a package.json in root for you$ npm listlists all installed packages$ npm pruneremoves packages not depended on by your project according to your package.json$ npm outdatedtells you which installed packages are outdated with respect to what is current in the npm registry but allowable by the version definition in your package.json

Installing Modules using NPM$ npm install <Module Name>

example:

$ npm install express

● installed express module, it created node_modules directory in the current directory where it installed the express module.

Installing Packages in Global Mode$ npm install express -g

Installing a Specific Version of a Package

Uninstalling Local Packages

package.json

package.json (2)

npm run ...$ npm run test

$ npm run start

The Dependencies propertyThe dependencies property of a module's package.json is where dependencies - the other modules that this module uses - are defined. The dependencies property takes an object that has the name and version at which each dependency should be used

Version syntax~version Approximately equivalent to version^version Compatible with versionversion Must match version exactly>version Must be greater than version>=version <version<=version1.2.x 1.2.0, 1.2.1, etc., but not 1.3.0* Matches any versionlatest Obtains latest release

matches the most recent minor version (the middle number). ~1.2.3 will match all 1.2.x versions but will miss 1.3.0.

Matches the most recent major version (the first number). ^1.2.3 will match any 1.x.x release including 1.3.0, but will hold off on 2.0.0

The devDependencies propertyThe devDependencies property of a package.json is almost identical to the dependencies property in terms of structure, with a key difference. The dependencies property is used to define the dependencies that a module needs to run in production. The devDependencies property is usually used to define the dependencies the module needs to run in development

--save vs --save-dev$ npm install [package_name] --save

$ npm install [package_name] --save-dev

● --save-dev is used to save the package for development purpose. Example: unit tests, minification..

● --save is used to save the package required for the application to run.

thanks for your attention any questions?