Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability SAaNS: Introduction to Unix Grégory Mounié 2014-2015 1 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalab Outline Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability 2 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability 3 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalab whoami: Grégory Mounié • [email protected]• Associate professor at Ensimag • Office: Building of Inria Montbonnot (11 km of the campus) • Current teaching: mainly Operating Systems; Networks and distributed systems; Python; • Former teaching: Networks, Java, Ada, Operational research, OpenGL • Research: Scheduling for High Performance Computing (HPC), or the art to put cubes in boxes 3 / 75
19
Embed
whoami: Grégory Mounié - ENSIMAG · PDF fileUnix® and Linux® System Administration Handbook,Fourth ... • ed (brotherof sed) ... •...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
SAaNS: Introduction to Unix
Grégory Mounié
2014-2015
1 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Outline
Presentation of the lecture
History of unix systems
Base concepts
Linux installation
CLI
Configuration
Distribution and Scalability
2 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Presentation of the lecture
History of unix systems
Base concepts
Linux installation
CLI
Configuration
Distribution and Scalability
3 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
whoami: Grégory Mounié
• [email protected]• Associate professor at Ensimag• Office: Building of Inria Montbonnot (11 km of the campus)• Current teaching: mainly Operating Systems; Networks anddistributed systems; Python;
• Former teaching: Networks, Java, Ada, Operational research,OpenGL
• Research: Scheduling for High Performance Computing(HPC), or the art to put cubes in boxes
3 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Content of the course
• OS basics and OS administration in 3 hours• Network basics and Network administration in 3 hours• Security in 3 hours• Perl in (3 + 3 hours)
4 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Bibliography
Main referenceUnix® and Linux® System Administration Handbook, FourthEdition, Video Enhanced Edition.Evi Nemeth; Garth Snyder; Trent R. Hein; Ben Whaley
Basics
OSModern Operating Systems, Tanenbaum
NetworkComputer Networks, Tanenbaum
Network and system programmingUnix programming, Stevens (a bit old but still the reference formany low levels and raw stuff)
5 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Read the man, Luke !
• Man pages• GNU Texinfo (Html with automatic index)• Package-specific doc
• typically in /usr/share/doc/• RFC and Internet documents
• beware of version mismatch
6 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Challenges
• People with sufficent background may skip the basics part ofthe lecture and do the challenge instead.
Challenge for the OS lectureYou are employee of X and you will go to Paques Island to sellsome Y. The major provider of Y is an US society. You will crossthe US border with commercial documents in your laptop.Choose a way to protect your documents, implement it and presentit to others lecturers (10 min at the end of the lecture)
7 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Duties of the system administrator
• Account provisioning• Adding and removing hardware• Performing backups• Installing and upgrading software• Monitoring the system• Troubleshooting• Maintaining local documentation• Vigilantly monitoring security• Fire fighting
Difficult pointEach point is simple. Complexity come from the cross-referencesbetween points.
8 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Origin of UNIX
in the beginning was multics (65-68)
• failure ⇒ Bell decides to stop making operating systems• Kernigham and Ritchie et al. develop a small operating systemof their own
• by derision they call it UNICS (69)• and create a programming language (C) to recode it
• property of Bell Labs• restrained diffusion when successful
9 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
History of UNIX
• sources was given (Unix V6) to users (device drivers)
Berkeley University (76)
• first version of BSD• large diffusion in universities
Parallel evolution, in 1983
• 4.3BSD (SunOS)• Unix System V
10 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
History of UNIX
• no free (as in speech) version• development of internet:
• ⇒ ideal conditions for the development of free unices:• strong demand• skilled peoples• means enabling common development
11 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
MINIX and Linux
Philosophy
• Minix (85) written by Prof. A.Tannenbaum for his courses
• internet ⇒ large diffusion andimprovements
• but restriction on user rights• Linux written by a student LinusTorvalds
• support of recent architectures• GPL licence
12 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Unixes Today
1. Linux and its embedded variants:• Android, Firefox OS, Sailfish OS (ex-Mer, ex-Meego), Ubuntu
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Computer are fast, stupid and huge but we have to workwith
Computer are:• stupid [G. Berry]:: only know the 4 operations in base 2 anddata movement
fast do 1 operation while a photon travels few centimetershuge The number of bits in a disk is similar to the solar mass of our
galaxy
Computer science challengeHow to build a non-stupid result using numerous simple operations?
Computer science 4 pillars [G. Doweq]
1. Algorithmic :: Express human idea in computer logic2. Data :: data structure and storage to store human ideas
efficiently for the computer logic3. Compilation :: programming languages to explain what to do
to the computer (fast, correct, easy to understand, etc. )4. Hardware :: How to build a computer
17 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Are we using 40 years old tools ???
Yes some tools worked on dot-matrix printer before screenage:
• ed (brother of sed) line-by-line text editor is stillinstalled in MacOS X, and vi is installed inbusybox used by many embedded systems(Android phones, DSLboxes, linux-base networkrouter)
No git and vlc are 10 years old
18 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Does Unix 40 year history means. . .
• we are using 40 year old tools ?
• Most concepts are present in Multics• Some tools are still the same• however, many tools evolved• time filters things : only the mostvaluable things are kept over time
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Unix concepts
composition and K.I.S.S • lots of small and efficient tools• easy composition of different tools to obtainpowerful effects
modularity Unix =• a kernel handling low level tasks• a set of applications
(almost) everything is a file • sockets, devices, pipes, . . .• uniform way of handling all resources
20 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Everything is a file
• Why ?
In computers, all devices are about IO
• reading data from a device• writing data to a device
• File interface for all device ⇒ Fewer system calls
21 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
System Calls
22 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Windows vs Linux http server
23 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Multitasking
• an executing program is called a process• several processes execute simultaneously
• sharing of memory (code and data)• sharing of CPU resources• sharing of peripheral devices
Warningprocess != program. A process is a program in execution
24 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Controlling process
Process attributes
• PID unique process ID number• PPID, UID, GID, priority
Process control
• ps, kill, nice• strace, gdb• /proc filesystem
25 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Multi-user
• several users on the same machine• notion of user accounts• home directory, ID number (UID), password, name, . . .
• each file is owned by one of the users• processes are owned by the user launching them• several processes belonging to different users coexist in thesame system
26 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Rights
• Users have credentials to manipulate the files (everything is afile)Basic model user/group/other: chmod, setuidAdvanced model ACL
Basic security ruleLimit the credentials to the minimum (It is a lot of work)The system may refuse some actions even with administrativerights (security, DRM, containment)
27 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Memory handling
• two types of memory :• physical memory : the one really existing inside the machine• virtual memory : the address space used by processes
• less physical than virtual• if physical becomes overused ⇒ swapping on disk• laziness : memory is physically reserved at the first r/w
• hardware translation from virtual to physical (MMU)• software management of hardware translation (Page Table)
28 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Filesystem
• logical organization (tree) of physical space• dependent on the OS• numerous formats• DOS/Windows 98 (FAT16, FAT32), Windows NT and later
• a partition has to be mounted to become accessible
Example (mount)mount (1) to manage mounted partitionsAllow to use multiple partitions and hardwares in a single tree.
29 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Low level: filesystem is a block array
• Hardware space is a large block array• Every normal file data are stored in a particular sequence ofblock
• The sequence of block of a file is described in a structure(inode) stored in a (fixed size) table, with the file metadata(UID, access right . . . save the file name)
• A directory is just a file (a single block or a sequence) storingpairs of (file name, inode number)
Example (stat)stat (1) / stat (2) to get information on a particular file
30 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Medium Level: filesystem is an hardware independant
• All filesystems expose the same interface thus similar conceptas the inheritance of object programming may be applied:
• Programs and systems use filesytem through the same genericinterface (VFS)
• Filesystems differences are hidden from common users• Matroska stack for network filesystems: NFS expose VFS
interface and access (remotely) the data through VFS)
31 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
High level: Main directories
standard rules
/bin /sbin base system programs/usr/bin programs
/usr/share data files for programs in /usr/bin/usr/share/doc program and system documentation
/home user home directories/var /var/log log files, changing system files
/tmp temporary files/etc configuration files
/usr/local/ locally compiled or installed file
32 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Special files
• Directories• Character (eg. mouse) and Block (HD, SSD) device files• Sockets• Named pipes• Symbolic link
33 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
• process communication and synchronization• energy saving
34 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Linux distributions
• Distribution = a set of programs distributed with the kernel• base tools for OS configuration and maintenance• user applications
• Distribution = hierarchy of directories• file positions• configuration files• one (loose) norm : FHS (Filesystem Hierarchy Standard)
35 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Main Desktop Linux distributions
• Debian (Ubuntu, Mint)• Redhat (Fedora, RHEL, CentOS, OpenSuse)• Mageïa (ex Mandrake, ex Mandriva)• Oracle Linux (ex Netware)• Gentoo• Arch linux• Slackware There are 300+ others http://distrowatch.com.
36 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Counter-example: Android
• The kernel is the same• The set of program is very different:
• JVM (Dalvik): most programs are Java program• C library (Bionic, not the full blend GNU Library)• Some other ad-hoc daemons
37 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Hardware handling
two types of hardware :
1. handled by the kernel• serial port, disk controllers, keyboard, . . .
2. handled partially by applications (user-space drivers)• graphical card, some USB gadgets, . . .
golden rules for System Administration
• no material too recent• lots of online documentation to read first
38 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Different steps
• preparation• boot• disk partitioning• files installations• base system configuration• accounts creations
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Preparation
• check if the disk can be erased• else re-partitioning• boot on: CD/DVD, USB key, network (eg. PXE), harddisk/SSD, floppy disk
• UEFI may stop you from booting anything but signed kernel• GPT is much modern alternative than standard PC partitioning• Raid (hard/soft) and LVM helps a lot but should be used withcaution
40 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Disk partitioning
• swap partition• / : main system partition• /boot : eventually• /home : users files• /var : changing system files• /tmp : temporary files
Why so many partitions ?Containing problems in case of filled-up partitions (eg. /var/log/)
41 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Files installations
• choice of type of installation• desktop• workstation• server• scientific computing• graphic station• media center• . . .
• more fine tune by manual packages selection
42 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Why so many filesystem choices
The needs change with time: storage increase with time, or changetechnology.
• Ext4 is still the standard for desktop and phone• ZFS (Sun/Oracle) allows plenty of nice things (integrity,storage pool, snapshot, copy-on-write, deduplication) but isnot part of the standard linux kernel and distribution (FUSE)
• Btrfs is still not ready yet, but should arrive very soon
43 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Packages
Definition (Package)application / library / set of applications
• add or removes packages instead of files• goal : ease administration• handling of dependencies between packages (The maindifference with GooglePlay, AppleStore, etc.)
44 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Different packages systems
• DEB : debian, ubuntu, mint. . .• RPM : redhat, mandriva, suse, . . .• TAR : just an archive file without dependencies• some conversion tools between packages (alien)
45 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
User accounts
• Role-based: at least two categories of accounts• administrators : root• user accounts
• possibilities to delegate root powers to standard users (setuid,sudo)
Warningthe root account should only be used for administration
46 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Account creations
• creating a new user account• username, home directory, password• all users are referenced in the /etc/passwd file• encrypted passwords are to be found in the /etc/shadow file
• adduser command (useradd: low level)
47 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Advanced account management
PAM: Pluggable Authentication Modules (Debian) libpam has 66modules. Most are different ways to authenticate auser (password, ssh keys, kerberos, two-factor, ldap,mysql, geoIP, etc.)
Kerberos third party cryptographic authenticationACL Access control lists fine grain chmod on filesroot account strong password, indirect login (su/sudo), lost root
passwordPOSIX capabilities program/shell may withdraw some rightsLDAP and Active Directory centralizing account management
48 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Configuration files
• configuration files are text files• easy to modify using any text editor• global configuration files in /etc• local configuration files hidden in home directory• one file for every application
49 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Access rights
• each file has an owner :• initially, the file creator
• rights are associated to each file• reading, writing, execution rights
• for the owner• for the members of the group• for all other users
50 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Groups
• each user belongs to a one or more groups• possibility to share files among all users of a group• security and flexibility• the list of groups is in /etc/group• the root user can create new groups
51 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Mounting points
• under DOS/WINDOWS : different peripheral with different filesystems
• a: c: d: . . .• under linux : only one root directory (/)
• adding new devices with the mount command
• configuration file : /etc/fstab
52 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Command line
• text based interface to the system• one main disadvantage : less intuitive than mouse interface• many advantages :
• scripts to automate actions• very useful for distant administration (ssh)• same interface everywhere• powerful for mass operations (e.g. renaming 200 files)• very quick : automatic completion• completion allows to avoid typing mistakes
53 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Base commands
ls list files in directorycat display a filerm remove a filecd change directory
mkdir make new directorymv move a filecp copy a file
grep filter file contentxargs take input as argument of the following command
54 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
sort sortuniq remove consecutive duplicatetee save pipe in a filewc word count
head and tail first lines and last lines of a filebasename remove suffix in a name
55 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Script internal commands
• Basic control flow:• branch (if-then-else)• loops• arrays• regular expressions• matching• literal and special character
• The syntax varies with the interpreters• sh/bash/dash, csh/tcsh, ksh/zsh
56 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Example
Simple command, complex interactionFind the number of line of the 5 largest files in the current directoryand its sub-directoriesfind . -type f -exec ls -l ’{}’ ’;’ | sort -k 5 |tail -5 | xargs wc -l
57 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Documentation
• all commands are programs• manual pages : man
• other doc in /usr/share/doc :
• programs documentation• HOWTO files
58 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Devices
• all devices are accessible as files in the /dev directory• access rights to devices == standard files rights• special files created using the mknod command• standard commands (cat, . . . ) working !• example : /dev/mouse
59 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
• SCSI and SATA disks• /dev/sda, /dev/sdb, . . .• first partition on first SCSI : /dev/sda1
• SDcard disks (Android)• /dev/block/mmcblk0
61 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Configuration files
• all configuration in text files• allow easy automatic modifications using scripts• almost all files in /etc• there exists some graphical frontends and global managementsystems
WarningConfiguration files vary from distribution to distribution as dofrontends.
62 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Main configuration files
• /etc/fstab : all mount points• /etc/modules.conf : kernel modules configuration• /etc/X11/xorg.conf : X11 configuration file• /etc/cups/ : all config files for CUPS service• /etc/resolv.conf : DNS config (often automatically generated)• . . .
63 / 75
Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability
Services
• service : functionality offered by the machine• program running in the background : daemon• examples :