Top Banner

of 447

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • Android System Development

    Android SystemDevelopment

    Free Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Latest update: June 11, 2014.Document updates and sources:http://free-electrons.com/doc/training/android

    Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 1/447

  • Rights to copy

    Copyright 2004-2014, Free ElectronsLicense: Creative Commons Attribution - Share Alike 3.0http://creativecommons.org/licenses/by-sa/3.0/legalcode

    You are free:

    I to copy, distribute, display, and perform the workI to make derivative worksI to make commercial use of the work

    Under the following conditions:

    I Attribution. You must give the original author credit.I Share Alike. If you alter, transform, or build upon this work, you may distribute

    the resulting work only under a license identical to this one.

    I For any reuse or distribution, you must make clear to others the license terms ofthis work.

    I Any of these conditions can be waived if you get permission from the copyrightholder.

    Your fair use and other rights are in no way affected by the above.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 2/447

  • Electronic copies of these documents

    I Electronic copies of your particular version of the materials areavailable on:http://free-electrons.com/doc/training/android

    I Open the corresponding documents and use them throughoutthe course to look for explanations given earlier by theinstructor.

    I You will need these electronic versions because we neitherprint any index nor any table of contents (quite highenvironmental cost for little added value)

    I For future reference, see the first slide to see where documentupdates will be available.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 3/447

  • Hyperlinks in the document

    There are many hyperlinks in the document

    I Regular hyperlinks:http://kernel.org/

    I Kernel documentation links:Documentation/kmemcheck.txt

    I Links to kernel source files and directories:drivers/input

    include/linux/fb.h

    I Links to the declarations, definitions and instances of kernelsymbols (functions, types, data, structures):platform_get_irq()

    GFP_KERNEL

    struct file_operations

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 4/447

  • Free Electrons at a glance

    I Engineering company created in 2004(not a training company!)

    I Locations: Orange, Toulouse, Saint Etienne / Lyon (France)

    I Serving customers all around the worldSee http://free-electrons.com/company/customers/

    I Head count: 9Only Free Software enthusiasts!

    I Focus: Embedded Linux, Linux kernel, Android Free Software/ Open Source for embedded and real-time systems.

    I Activities: development, training, consulting, technicalsupport.

    I Added value: get the best of the user and developmentcommunity and the resources it offers.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 5/447

  • Free Electrons on-line resources

    I All our training materials:http://free-electrons.com/docs/

    I Technical blog:http://free-electrons.com/blog/

    I Quarterly newsletter:http://lists.free-

    electrons.com/mailman/listinfo/newsletter

    I News and discussions (Google +):https://plus.google.com/+FreeElectronsDevelopers

    I News and discussions (LinkedIn):http://linkedin.com/groups/Free-Electrons-4501089

    I Quick news (Twitter):http://twitter.com/free_electrons

    I Linux Cross Reference - browse Linux kernel sources on-line:http://lxr.free-electrons.com

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 6/447

  • Mailing lists for session participants

    If you participated to a course taught by Free Electrons, you cansubscribe to our mailing lists dedicated to our training customers:

    I To ask questions about the topics covered by the courses, andto get a first level of technical support for issues that your arefacing in real life or with your own hardware.

    I To share your experience with other people having taken thesame course and done the same practical labs.

    I To share further comments and suggestions about coursecontents.

    I To get news about important updates to the course materials.

    See http://free-electrons.com/training/mailing-listsA simple way to register: fill the form on http://j.mp/1r1HhkZ

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 7/447

  • Generic course information

    Generic courseinformationFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 8/447

  • Hardware used in this training session

    BeagleBone Black, from CircuitCo

    I Texas Instruments AM335x (ARM Cortex-A8)I Powerful CPU, with 3D acceleration,

    additional processors (PRUs) and lots ofperipherals.

    I 512 MB of RAMI 2 GB of on-board eMMC storage

    (4 GB in Rev C)

    I USB host and USB device portsI microSD slotI HDMI portI 2 x 46 pins headers, with access to many

    expansion buses (I2C, SPI, UART and more)

    I A huge number of expansion boards, calledcapes. See http://beagleboardtoys.com/.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 9/447

  • Course outline - Day 1

    Building Android

    I Introduction to Android

    I Getting Android sources

    I Building and booting Android

    I Introduction to the Linux kernel

    I Compiling and booting the Linux kernel

    Labs: download Android sources, compile them and boot themwith the Android emulator. Recompile the Linux kernel.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 10/447

  • Course outline - Day 2

    Android kernel, boot and filesystem details

    I Android changes to the Linux kernel

    I Android bootloaders

    I Booting Android

    I Using ADB

    I Android filesystem

    Labs: customize, compile and boot Android for the BeagleBoneBlack board.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 11/447

  • Course outline - Day 3

    Supporting a new product and customizing it

    I Android build system. Add a new module and product.

    I Android native layer - Bionic, Toolbox, init, various daemons,Dalvik, hardware abstraction, JNI...

    Labs: Use ADB, create a new product, customize the product forthe BeagleBone Black board.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 12/447

  • Course outline - Day 4

    Android framework and applications

    I Android framework for applications

    I Introduction to application development

    I Android packages

    I Advise and resources

    Labs: compile an external library and a native application tocontrol a USB missile launcher. Create a JNI library and developan Android application to control the device.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 13/447

  • Participate!

    During the lectures...

    I Dont hesitate to ask questions. Other people in the audiencemay have similar questions too.

    I This helps the trainer to detect any explanation that wasntclear or detailed enough.

    I Dont hesitate to share your experience, for example tocompare Linux / Android with other operating systems usedin your company.

    I Your point of view is most valuable, because it can be similarto your colleagues and different from the trainers.

    I Your participation can make our session more interactive andmake the topics easier to learn.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 14/447

  • Practical lab guidelines

    During practical labs...

    I We cannot support more than 8 workstations at once (eachwith its board and equipment). Having more would make thewhole class progress slower, compromising the coverage of thewhole training agenda (exception for public sessions: up to 10people).

    I So, if you are more than 8 participants, please form up to 8working groups.

    I Open the electronic copy of your lecture materials, and use itthroughout the practical labs to find the slides you need again.

    I Dont copy and paste from the PDF slides.The slides contain UTF-8 characters that look the same asASCII ones, but wont be understood by shells or compilers.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 15/447

  • Cooperate!

    As in the Free Software and Open Source community, cooperationduring practical labs is valuable in this training session:

    I If you complete your labs before other people, dont hesitateto help other people and investigate the issues they face. Thefaster we progress as a group, the more time we have toexplore extra topics.

    I Explain what you understood to other participants whenneeded. It also helps to consolidate your knowledge.

    I Dont hesitate to report potential bugs to your instructor.

    I Dont hesitate to look for solutions on the Internet as well.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 16/447

  • Command memento sheet

    I This memento sheet givescommand examples for the mosttypical needs (looking for files,extracting a tar archive...)

    I It saves us 1 day of UNIX / Linuxcommand line training.

    I Our best tip: in the command lineshell, always hit the Tab key tocomplete command names and filepaths. This avoids 95% of typingmistakes.

    I Get an electronic copy onhttp://free-electrons.com/

    doc/training/embedded-

    linux/command_memento.pdf

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 17/447

  • vi basic commands

    I The vi editor is very useful tomake quick changes to files in anembedded target.

    I Though not very user friendly atfirst, vi is very powerful and itsmain 15 commands are easy tolearn and are sufficient for 99% ofeveryones needs!

    I Get an electronic copy onhttp://free-electrons.com/

    doc/training/embedded-

    linux/vi_memento.pdf

    I You can also take the quick tutorialby running vimtutor. This is aworthy investment!

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 18/447

  • Practical lab - Training Setup

    Prepare your lab environment

    I Download the lab archive

    I Enforce correct permissions

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 19/447

  • Introduction to Android

    Introduction toAndroidFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 20/447

  • Introduction to Android

    Features

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 21/447

  • Features

    I All you can expect from a modern mobile OS:I Application ecosystem, allowing to easily add and remove

    applications and publish new features across the entire systemI Support for all the web technologies, with a browser built on

    top of the well-established WebKit rendering engineI Support for hardware accelerated graphics through OpenGL ESI Support for all the common wireless mechanisms: GSM,

    CDMA, UMTS, LTE, Bluetooth, WiFi.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 22/447

  • Introduction to Android

    History

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 23/447

  • Early Years

    I Began as a start-up in Palo Alto, CA, USA in 2003

    I Focused from the start on software for mobile devices

    I Very secretive at the time, even though founders achieved alot in the targeted area before founding it

    I Finally bought by Google in 2005

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 24/447

  • Opening Up

    I Google announced the Open Handset Alliance in 2007, aconsortium of major actors in the mobile area built aroundAndroid

    I Hardware vendors: Intel, Texas Instruments, Qualcomm,Nvidia, etc.

    I Software companies: Google, eBay, etc.I Hardware manufacturers: Motorola, HTC, Sony Ericsson,

    Samsung, etc.I Mobile operators: T-Mobile, Telefonica, Vodafone, etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 25/447

  • Android Open Source Project (AOSP)

    I At every new version, Google releases its source code throughthis project so that community and vendors can work with it.

    I One major exception: Honeycomb has not been releasedbecause Google stated that its source code was not cleanenough to release it.

    I One can fetch the source code and contribute to it, eventhough the development process is very locked by Google

    I Only a few devices are supported through AOSP though, onlythe two most recent Android development phones and tablets(part of the Nexus brand) and the pandaboard

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 26/447

  • Android Releases

    I Each new version is given a dessert name

    I Released in alphabetical orderI Latest releases:

    I Android 2.3 GingerbreadI Android 3.X HoneycombI Android 4.0 Ice Cream SandwichI Android 4.1/4.2/4.3 Jelly BeanI Android 4.4 KitKat

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 27/447

  • Android Versions

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 28/447

  • Introduction to Android

    Architecture

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 29/447

  • Architecture

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 30/447

  • The Linux Kernel

    I Used as the foundation of the Android system

    I Numerous additions from the stock Linux, including new IPC(Inter-Process Communication) mechanisms, alternativepower management mechanism, new drivers and variousadditions across the kernel

    I These changes are beginning to go into the staging/ area ofthe kernel, as of 3.3, after being a complete fork for a longtime

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 31/447

  • Android Libraries

    I Gather a lot of Android-specific libraries to interact at alow-level with the system, but third-parties libraries as well

    I Bionic is the C library, SurfaceManager is used for drawingsurfaces on the screen, etc.

    I But also WebKit, SQLite, OpenSSL coming from the freesoftware world

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 32/447

  • Android Runtime

    Handles the execution of Android applications

    I Almost entirely written from scratch by Google

    I Contains Dalvik, the virtual machine that executes everyapplication that you run on Android, and the core library forthe Java runtime, coming from Apache Harmony project

    I Also contains system daemons, init executable, basic binaries,etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 33/447

  • Android Framework

    I Provides an API for developers to create applications

    I Exposes all the needed subsystems by providing an abstraction

    I Allows to easily use databases, create services, expose data toother applications, receive system events, etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 34/447

  • Android Applications

    I AOSP also comes with a set of applications such as the phoneapplication, a browser, a contact management application, anemail client, etc.

    I However, the Google apps and the Android Market app arentfree software, so they are not available in AOSP. To obtainthem, you must contact Google and pass a compatibility test.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 35/447

  • Practical lab - Android Source Code

    I Install all the developmentpackages needed to fetch andcompile Android

    I Download the repo utility

    I Use repo to download the sourcecode for Android and for all itscomponents

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 36/447

  • Android Source Code and Compilation

    Android SourceCode andCompilationFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 37/447

  • Android Source Code and Compilation

    How to get the source code

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 38/447

  • Source Code Location

    I The AOSP project is available athttp://source.android.com

    I On this site, along with the code, you will find some resourcessuch as technical details, how to setup a machine to buildAndroid, etc.

    I The source code is split into several Git repositories forversion control. But as there is a lot of source code, a singleGit repository would have been really slow

    I Google split the source code into a one Git repository percomponent

    I You can easily browse these git repositories usinghttps://code.google.com/p/android-source-

    browsing/source/browse/

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 39/447

  • Source code licenses

    I Mostly two kind of licenses:I GPL/LGPL Code: LinuxI Apache/BSD: All the restI In the external folder, it depends on the component

    I While you might expect Googles apps for Android, like theAndroid Market (now called Google Play Store), to be in theAOSP as well, these are actually proprietary and you need tobe approved by Google to get them.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 40/447

  • Repo

    I This makes hundreds of Git repositories

    I To avoid making it too painful, Google also created a tool:repo

    I Repo aggregates these Git repositories into a single folderfrom a manifest file describing how to find these and how toput them together

    I Also aggregates some common Git commands such as diff orstatus that are run across all the Git repositories

    I You can also execute a shell command in each repositorymanaged by Repo using the repo forall command

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 41/447

  • Repos manifest

    I repo relies on a git repository that will contain XML filescalled manifests

    I These manifests gives the information about where todownload some source code and where to store it. It can alsoprovide some additional and optional information such as arevision to use, an alternative server to download from, etc.

    I The main manifests are stored in this git repo, and are sharedbetween all the users, but you can add some local manifests.

    I repo will also use any XML file that is under.repo/local_manifests

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 42/447

  • Manifests syntax

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 43/447

  • Android Source Code and Compilation

    Source code organization

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 44/447

  • Source Code organization 1/3

    I Once the source code is downloaded, you will find severalfolders in it

    bionic/ is where Androids standard C library is stored

    bootable/ contains code samples regarding the boot of anAndroid device. In this folder, you will find theprotocol used by all Android bootloaders and arecovery image

    build/ holds the core components of the build system

    cts/ The Compatibility Test Suite

    dalvik/ contains the source code of the Dalvik virtualmachine

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 45/447

  • Source Code Organization 2/3

    development/ holds the development tools, debug applications,API samples, etc

    device/ contains the device-specific components

    docs/ contains HTML documentation hosted athttp://source.android.com

    external/ is one of the largest folders in the source code, itcontains all the external projects used in the Androidcode

    frameworks/ holds the source code of the various parts of theframework

    hardware/ contains all the hardware abstraction layers

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 46/447

  • Source Code Organization 3/3

    libcore/ is the Java core library

    libnativehelper/ contains a few JNI helpers for the Android baseclasses

    ndk/ is the place where you will find the NativeDevelopment Kit, which allows to build nativeapplications for Android

    packages/ contains the standard Android applications

    prebuilt/ holds all the prebuilt binaries, most notably thetoolchains

    sdk/ is where you will find the Software Development Kit

    system/ contains all the basic pieces of the Android system:init, shell, the volume manager, etc.

    I You can get a more precise description athttp://elinux.org/Master-android

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 47/447

  • Android Source Code and Compilation

    Compilation

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 48/447

  • Android Compilation Process

    I Androids build system relies on the well-tried GNU/Makesoftware

    I Android is using a product notion which corresponds to thespecifications of a shipping product, i.e. crespo for the GoogleNexus S vs crespo4g for the Sprints Nexus S with LTEsupport

    I To start using the build system, you need to include the filebuild/envsetup.sh that defines some useful macros forAndroid development or sets the PATH variable to include theAndroid-specific commands

    I source build/envsetup.sh

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 49/447

  • Prepare the process

    I Now, we can get a list of all the products available and selectthem with the lunch command

    I lunch will also ask for a build variant, to choose betweeneng, user and userdebug, which corresponds to which kindof build we want, and which packages it will add

    I You can also select variants by passing directly the comboproduct-variant as argument to lunch

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 50/447

  • Compilation

    I You can now start the compilation just by running make

    I This will run a full build for the currently selected product

    I There are many other build commands:

    make Builds only the package, instead of goingthrough the entire build

    make clean Cleans all the files generated by previouscompilations

    make clean- Removes all the files generated bythe compilation of the given package

    mm Builds all the modules in the current directorymmm builds all the modules in the given

    directory

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 51/447

  • Android Source Code and Compilation

    Contribute

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 52/447

  • Gerrit

    I For the Android development process, Google also developeda tool to manage projects and ease code reviews.

    I It once again uses Git to do so and Repo is also built aroundit so that you can easily contribute to Android

    I To do so, start a new branch withrepo start

    I Do your usual commits with Git

    I When you are done, upload to Gerrit using repo upload

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 53/447

  • Practical lab - First Compilation

    I Configure which system to buildAndroid for

    I Compile your first Android rootfilesystem

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 54/447

  • Linux kernel introduction

    Linux kernelintroductionFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 55/447

  • Linux kernel introduction

    Linux features

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 56/447

  • History

    I The Linux kernel is one component of a system, which alsorequires libraries and applications to provide features to endusers.

    I The Linux kernel was created as a hobby in 1991 by a Finnishstudent, Linus Torvalds.

    I Linux quickly started to be used as the kernel for free softwareoperating systems

    I Linus Torvalds has been able to create a large and dynamicdeveloper and user community around Linux.

    I Nowadays, more than one thousand people contribute to eachkernel release, individuals or companies big and small.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 57/447

  • Linux kernel in the system

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 58/447

  • Linux kernel main roles

    I Manage all the hardware resources: CPU, memory, I/O.

    I Provide a set of portable, architecture and hardwareindependent APIs to allow user space applications andlibraries to use the hardware resources.

    I Handle concurrent accesses and usage of hardwareresources from different applications.

    I Example: a single network interface is used by multiple userspace applications through various network connections. Thekernel is responsible to multiplex the hardware resource.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 59/447

  • Inside the Linux kernel

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 60/447

  • Linux license

    I The whole Linux sources are Free Software released under theGNU General Public License version 2 (GPL v2).

    I For the Linux kernel, this basically implies that:I When you receive or buy a device with Linux on it, you should

    receive the Linux sources, with the right to study, modify andredistribute them.

    I When you produce Linux based devices, you must release thesources to the recipient, with the same rights, with norestriction.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 61/447

  • Linux kernel key features

    I Portability and hardwaresupport. Runs on mostarchitectures.

    I Scalability. Can run onsuper computers as well ason tiny devices (4 MB ofRAM is enough).

    I Compliance to standardsand interoperability.

    I Exhaustive networkingsupport.

    I Security. It cant hide itsflaws. Its code is reviewedby many experts.

    I Stability and reliability.

    I Modularity. Can includeonly what a system needseven at run time.

    I Easy to program. You canlearn from existing code.Many useful resources onthe net.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 62/447

  • Supported hardware architectures

    I See the arch/ directory in the kernel sources

    I Minimum: 32 bit processors, with or without MMU, and gccsupport

    I 32 bit architectures (arch/ subdirectories)Examples: arm, avr32, blackfin, c6x, m68k, microblaze,mips, score, sparc, um

    I 64 bit architectures:Examples: alpha, arm64, ia64, tile

    I 32/64 bit architecturesExamples: powerpc, x86, sh, sparc

    I Find details in kernel sources: arch//Kconfig,arch//README, or Documentation//

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 63/447

  • System calls

    I The main interface between the kernel and user space is theset of system calls

    I About 300 system calls that provide the main kernel servicesI File and device operations, networking operations,

    inter-process communication, process management, memorymapping, timers, threads, synchronization primitives, etc.

    I This interface is stable over time: only new system calls canbe added by the kernel developers

    I This system call interface is wrapped by the C library, anduser space applications usually never make a system calldirectly but rather use the corresponding C library function

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 64/447

  • Virtual filesystems

    I Linux makes system and kernel information available in userspace through virtual filesystems.

    I Virtual filesystems allow applications to see directories andfiles that do not exist on any real storage: they are created onthe fly by the kernel

    I The two most important virtual filesystems areI proc, usually mounted on /proc:

    Operating system related information (processes, memorymanagement parameters...)

    I sysfs, usually mounted on /sys:Representation of the system as a set of devices and buses.Information about these devices.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 65/447

  • Linux kernel introduction

    Linux versioning scheme anddevelopment process

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 66/447

  • Until 2.6 (1)

    I One stable major branch every 2 or 3 yearsI Identified by an even middle numberI Examples: 1.0.x, 2.0.x, 2.2.x, 2.4.x

    I One development branch to integrate new functionalities andmajor changes

    I Identified by an odd middle numberI Examples: 2.1.x, 2.3.x, 2.5.xI After some time, a development version becomes the new base

    version for the stable branch

    I Minor releases once in while: 2.2.23, 2.5.12, etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 67/447

  • Until 2.6 (2)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 68/447

  • Changes since Linux 2.6

    I Since 2.6.0, kernel developers have been able to introducelots of new features one by one on a steady pace, withouthaving to make disruptive changes to existing subsystems.

    I Since then, there has been no need to create a newdevelopment branch massively breaking compatibility with thestable branch.

    I Thanks to this, more features are released to users at afaster pace.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 69/447

  • 3.x stable branch

    I From 2003 to 2011, the official kernel versions were named2.6.x.

    I Linux 3.0 was released in July 2011I This is only a change to the numbering scheme

    I Official kernel versions are now named 3.x (3.0, 3.1, 3.2,etc.)

    I Stabilized versions are named 3.x.y (3.0.2, 3.4.3, etc.)I It effectively only removes a digit compared to the previous

    numbering scheme

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 70/447

  • New development model

    Using merge and bug fixing windows

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 71/447

  • New development model - Details

    I After the release of a 3.x version (for example), a two-weeksmerge window opens, during which major additions aremerged.

    I The merge window is closed by the release of test version3.(x+1)-rc1

    I The bug fixing period opens, for 6 to 10 weeks.

    I At regular intervals during the bug fixing period,3.(x+1)-rcY test versions are released.

    I When considered sufficiently stable, kernel 3.(x+1) isreleased, and the process starts again.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 72/447

  • More stability for the kernel source tree

    I Issue: bug and security fixes only releasedfor most recent stable kernel versions.

    I Some people need to have a recent kernel,but with long term support for securityupdates.

    I You could get long term support from acommercial embedded Linux provider.

    I You could reuse sources for the kernelused in Ubuntu Long Term Supportreleases (5 years of free security updates).

    I The http://kernel.org front pageshows which versions will be supported forsome time (up to 2 or 3 years), and whichones wont be supported any more(EOL: End Of Life)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 73/447

  • Whats new in each Linux release?

    I The official list of changes for each Linux release is just ahuge list of individual patches!commit aa6e52a35d388e730f4df0ec2ec48294590cc459Author: Thomas Petazzoni Date: Wed Jul 13 11:29:17 2011 +0200

    at91: at91-ohci: support overcurrent notification

    Several USB power switches (AIC1526 or MIC2026) have a digital outputthat is used to notify that an overcurrent situation is takingplace. This digital outputs are typically connected to GPIO inputs ofthe processor and can be used to be notified of those overcurrentsituations.

    Therefore, we add a new overcurrent_pin[] array in the at91_usbh_datastructure so that boards can tell the AT91 OHCI driver which pins areused for the overcurrent notification, and an overcurrent_supportedboolean to tell the driver whether overcurrent is supported or not.

    The code has been largely borrowed from ohci-da8xx.c andohci-s3c2410.c.

    Signed-off-by: Thomas Petazzoni Signed-off-by: Nicolas Ferre

    I Very difficult to find out the key changes and to get the globalpicture out of individual changes.

    I Fortunately, there are some useful resources availableI http://wiki.kernelnewbies.org/LinuxChangesI http://lwn.netI http://linuxfr.org, for French readers

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 74/447

  • Linux kernel introduction

    Kernel configuration

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 75/447

  • Kernel configuration and build system

    I The kernel configuration and build system is based onmultiple Makefiles

    I One only interacts with the main Makefile, present at thetop directory of the kernel source tree

    I Interaction takes placeI using the make tool, which parses the MakefileI through various targets, defining which action should be done

    (configuration, compilation, installation, etc.). Runmake help to see all available targets.

    I ExampleI cd linux-3.6.x/I make

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 76/447

  • Kernel configuration (1)

    I The kernel contains thousands of device drivers, filesystemdrivers, network protocols and other configurable items

    I Thousands of options are available, that are used toselectively compile parts of the kernel source code

    I The kernel configuration is the process of defining the set ofoptions with which you want your kernel to be compiled

    I The set of options dependsI On your hardware (for device drivers, etc.)I On the capabilities you would like to give to your kernel

    (network capabilities, filesystems, real-time, etc.)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 77/447

  • Kernel configuration (2)

    I The configuration is stored in the .config file at the root ofkernel sources

    I Simple text file, key=value style

    I As options have dependencies, typically never edited by hand,but through graphical or text interfaces:

    I make xconfig, make gconfig (graphical)I make menuconfig, make nconfig (text)I You can switch from one to another, they all load/save the

    same .config file, and show the same set of options

    I To modify a kernel in a GNU/Linux distribution: theconfiguration files are usually released in /boot/, togetherwith kernel images: /boot/config-3.2.0-31-generic

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 78/447

  • Kernel or module?

    I The kernel image is a single file, resulting from the linkingof all object files that correspond to features enabled in theconfiguration

    I This is the file that gets loaded in memory by the bootloaderI All included features are therefore available as soon as the

    kernel starts, at a time where no filesystem exists

    I Some features (device drivers, filesystems, etc.) can howeverbe compiled as modules

    I Those are plugins that can be loaded/unloaded dynamically toadd/remove features to the kernel

    I Each module is stored as a separate file in the filesystem,and therefore access to a filesystem is mandatory to usemodules

    I This is not possible in the early boot procedure of the kernel,because no filesystem is available

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 79/447

  • Kernel option types

    I There are different types of optionsI bool options, they are either

    I true (to include the feature in the kernel) orI false (to exclude the feature from the kernel)

    I tristate options, they are eitherI true (to include the feature in the kernel image) orI module (to include the feature as a kernel module) orI false (to exclude the feature)

    I int options, to specify integer valuesI hex options, to specify hexadecimal valuesI string options, to specify string values

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 80/447

  • Kernel option dependencies

    I There are dependencies between kernel options

    I For example, enabling a network driver requires the networkstack to be enabled

    I Two types of dependenciesI depends on dependencies. In this case, option A that depends

    on option B is not visible until option B is enabledI select dependencies. In this case, with option A depending

    on option B, when option A is enabled, option B isautomatically enabled

    I make xconfig allows to see all options, even those thatcannot be selected because of missing dependencies. In thiscase, they are displayed in gray

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 81/447

  • make xconfig

    make xconfig

    I The most common graphical interface to configure the kernel.

    I Make sure you readhelp -> introduction: useful options!

    I File browser: easier to load configuration files

    I Search interface to look for parameters

    I Required Debian / Ubuntu packages: libqt4-dev g++(libqt3-mt-dev for older kernel releases)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 82/447

  • make xconfig screenshot

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 83/447

  • make xconfig search interface

    Looks for a keyword in the parameter name. Allows to select orunselect found parameters.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 84/447

  • Kernel configuration options

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 85/447

  • Corresponding .config file excerpt

    Options are grouped by sections and are prefixed with CONFIG_.

    #

    # CD-ROM/DVD Filesystems

    #

    CONFIG_ISO9660_FS=m

    CONFIG_JOLIET=y

    CONFIG_ZISOFS=y

    CONFIG_UDF_FS=y

    CONFIG_UDF_NLS=y

    #

    # DOS/FAT/NT Filesystems

    #

    # CONFIG_MSDOS_FS is not set

    # CONFIG_VFAT_FS is not set

    CONFIG_NTFS_FS=m

    # CONFIG_NTFS_DEBUG is not set

    CONFIG_NTFS_RW=y

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 86/447

  • make gconfig

    make gconfig

    I GTK based graphicalconfiguration interface.Functionality similar to thatof make xconfig.

    I Just lacking a searchfunctionality.

    I Required Debian packages:libglade2-dev

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 87/447

  • make menuconfig

    make menuconfig

    I Useful when no graphics areavailable. Pretty convenienttoo!

    I Same interface found inother tools: BusyBox,Buildroot...

    I Required Debian packages:libncurses-dev

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 88/447

  • make nconfig

    make nconfig

    I A newer, similar textinterface

    I More user friendly (forexample, easier to accesshelp information).

    I Required Debian packages:libncurses-dev

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 89/447

  • make oldconfig

    make oldconfig

    I Needed very often!

    I Useful to upgrade a .config file from an earlier kernel release

    I Issues warnings for configuration parameters that no longerexist in the new kernel.

    I Asks for values for new parameters

    If you edit a .config file by hand, its strongly recommended torun make oldconfig afterwards!

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 90/447

  • Undoing configuration changes

    A frequent problem:

    I After changing several kernel configuration settings, yourkernel no longer works.

    I If you dont remember all the changes you made, you can getback to your previous configuration:$ cp .config.old .config

    I All the configuration interfaces of the kernel (xconfig,menuconfig, oldconfig...) keep this .config.old backupcopy.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 91/447

  • Configuration per architecture

    I The set of configuration options is architecture dependentI Some configuration options are very architecture-specificI Most of the configuration options (global kernel options,

    network subsystem, filesystems, most of the device drivers) arevisible in all architectures.

    I By default, the kernel build system assumes that the kernel isbeing built for the host architecture, i.e. native compilation

    I The architecture is not defined inside the configuration, but ata higher level

    I We will see later how to override this behaviour, to allow theconfiguration of kernels for a different architecture

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 92/447

  • Overview of kernel options (1)

    I General setupI Local version - append to kernel release allows to concatenate

    an arbitrary string to the kernel version that a user can getusing uname -r. Very useful for support!

    I Support for swap, can usually be disabled on most embeddeddevices

    I Configure standard kernel features (expert users) allows toremove features from the kernel to reduce its size. Powerful,but use with care!

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 93/447

  • Overview of kernel options (2)

    I Loadable module supportI Allows to enable or completely disable module support. If your

    system doesnt need kernel modules, best to disable since itsaves a significant amount of space and memory

    I Enable the block layerI If CONFIG_EXPERT is enabled, the block layer can be

    completely removed. Embedded systems using only raw flashstorage (MTD) can safely disable the block layer

    I Processor type and features (x86) or System type (ARM) orCPU selection (MIPS)

    I Allows to select the CPU or machine for which the kernel mustbe compiled

    I On x86, only optimization-related, on other architectures veryimportant since theres no compatibility

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 94/447

  • Overview of kernel options (3)

    I Kernel featuresI Tickless system, which allows to disable the regular timer tick

    and use on-demand ticks instead. Improves power savingsI High resolution timer support. By default, the resolution of

    timer is the tick resolution. With high resolution timers, theresolution is as precise as the hardware can give

    I Preemptible kernel enables the preemption inside the kernelcode (the user space code is always preemptible). See ourreal-time presentation for details

    I Power managementI Global power management option needed for all power

    management related featuresI Suspend to RAM, CPU frequency scaling, CPU idle control,

    suspend to disk

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 95/447

  • Overview of kernel options (4)

    I Networking supportI The network stackI Networking options

    I Unix sockets, needed for a form of inter-processcommunication

    I TCP/IP protocol with options for multicast, routing,tunneling, Ipsec, Ipv6, congestion algorithms, etc.

    I Other protocols such as DCCP, SCTP, TIPC, ATMI Ethernet bridging, QoS, etc.

    I Support for other types of networkI CAN bus, Infrared, Bluetooth, Wireless stack, WiMax stack,

    etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 96/447

  • Overview of kernel options (5)

    I Device driversI MTD is the subsystem for flash (NOR, NAND, OneNand,

    battery-backed memory, etc.)I Parallel port supportI Block devices, a few misc block drivers such as loopback,

    NBD, etc.I ATA/ATAPI, support for IDE disk, CD-ROM and tapes. A

    new stack existsI SCSI

    I The SCSI core, needed not only for SCSI devices but also forUSB mass storage devices, SATA and PATA hard drives, etc.

    I SCSI controller drivers

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 97/447

  • Overview of kernel options (6)

    I Device drivers (cont)I SATA and PATA, the new stack for hard disks, relies on SCSII RAID and LVM, to aggregate hard drives and do replicationI Network device support, with the network controller drivers.

    Ethernet, Wireless but also PPPI Input device support, for all types of input devices: keyboards,

    mice, joysticks, touchscreens, tablets, etc.I Character devices, contains various device drivers, amongst

    themI serial port controller driversI PTY driver, needed for things like SSH or telnet

    I I2C, SPI, 1-wire, support for the popular embedded busesI Hardware monitoring support, infrastructure and drivers for

    thermal sensors

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 98/447

  • Overview of kernel options (7)

    I Device drivers (cont)I Watchdog supportI Multifunction drivers are drivers that do not fit in any other

    category because the device offers multiple functionality at thesame time

    I Multimedia support, contains the V4L and DVB subsystems,for video capture, webcams, AM/FM cards, DVB adapters

    I Graphics support, infrastructure and drivers for framebuffersI Sound card support, the OSS and ALSA sound infrastructures

    and the corresponding driversI HID devices, support for the devices that conform to the HID

    specification (Human Input Devices)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 99/447

  • Overview of kernel options (8)

    I Device drivers (cont)I USB support

    I InfrastructureI Host controller driversI Device drivers, for devices connected to the embedded systemI Gadget controller driversI Gadget drivers, to let the embedded system act as a

    mass-storage device, a serial port or an Ethernet adapter

    I MMC/SD/SDIO supportI LED supportI Real Time Clock driversI Voltage and current regulatorsI Staging drivers, crappy drivers being cleaned up

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 100/447

  • Overview of kernel options (9)

    I For some categories of devices the driver is not implementedinside the kernel

    I PrintersI ScannersI Graphics drivers used by X.orgI Some USB devices

    I For these devices, the kernel only provides a mechanism toaccess the hardware, the driver is implemented in user space

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 101/447

  • Overview of kernel options (10)

    I File systemsI The common Linux filesystems for block devices: ext2, ext3,

    ext4I Less common filesystems: XFS, JFS, ReiserFS, GFS2, OCFS2,

    BtrfsI CD-ROM filesystems: ISO9660, UDFI DOS/Windows filesystems: FAT and NTFSI Pseudo filesystems: proc and sysfsI Miscellaneous filesystems, with amongst other flash filesystems

    such as JFFS2, UBIFS, SquashFS, cramfsI Network filesystems, with mainly NFS and SMB/CIFS

    I Kernel hackingI Debugging features useful for kernel developers

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 102/447

  • Linux kernel introduction

    Compiling and installing the kernelfor the host system

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 103/447

  • Kernel compilation

    I makeI in the main kernel source directoryI Remember to run multiple jobs in parallel if you have multiple

    CPU cores. Example: make -j 4I No need to run as root!

    I GeneratesI vmlinux, the raw uncompressed kernel image, at the ELF

    format, useful for debugging purposes, but cannot be bootedI arch//boot/*Image, the final, usually compressed,

    kernel image that can be bootedI bzImage for x86, zImage for ARM, vmImage.gz for Blackfin,

    etc.

    I arch//boot/dts/*.dtb, compiled Device Tree files(on some architectures)

    I All kernel modules, spread over the kernel source tree, as .kofiles.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 104/447

  • Kernel installation

    I make installI Does the installation for the host system by default, so needs

    to be run as root. Generally not used when compiling for anembedded system, and it installs files on the developmentworkstation.

    I InstallsI /boot/vmlinuz-

    Compressed kernel image. Same as the one inarch//boot

    I /boot/System.map-Stores kernel symbol addresses

    I /boot/config-Kernel configuration for this version

    I Typically re-runs the bootloader configuration utility to takethe new kernel into account.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 105/447

  • Module installation

    I make modules_installI Does the installation for the host system by default, so needs

    to be run as root

    I Installs all modules in /lib/modules//I kernel/

    Module .ko (Kernel Object) files, in the same directorystructure as in the sources.

    I modules.aliasModule aliases for module loading utilities. Example line:alias sound-service-?-0 snd_mixer_oss

    I modules.depModule dependencies

    I modules.symbolsTells which module a given symbol belongs to.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 106/447

  • Kernel cleanup targets

    I Clean-up generated files (to forcere-compilation):make clean

    I Remove all generated files. Needed whenswitching from one architecture to another.Caution: it also removes your .config file!make mrproper

    I Also remove editor backup and patch reject files(mainly to generate patches):make distclean

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 107/447

  • Linux kernel introduction

    Cross-compiling the kernel

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 108/447

  • Cross-compiling the kernel

    When you compile a Linux kernel for another CPU architecture

    I Much faster than compiling natively, when the target systemis much slower than your GNU/Linux workstation.

    I Much easier as development tools for your GNU/Linuxworkstation are much easier to find.

    I To make the difference with a native compiler, cross-compilerexecutables are prefixed by the name of the target system,architecture and sometimes library. Examples:mips-linux-gcc, the prefix is mips-linux-arm-linux-gnueabi-gcc, the prefix is arm-linux-gnueabi-

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 109/447

  • Specifying cross-compilation (1)

    The CPU architecture and cross-compiler prefix are defined throughthe ARCH and CROSS_COMPILE variables in the toplevel Makefile.

    I ARCH is the name of the architecture. It is defined by thename of the subdirectory in arch/ in the kernel sources

    I Example: arm if you want to compile a kernel for the armarchitecture.

    I CROSS_COMPILE is the prefix of the cross compilation toolsI Example: arm-linux- if your compiler is arm-linux-gcc

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 110/447

  • Specifying cross-compilation (2)

    Two solutions to define ARCH and CROSS_COMPILE:

    I Pass ARCH and CROSS_COMPILE on the make command line:make ARCH=arm CROSS_COMPILE=arm-linux- ...

    Drawback: it is easy to forget to pass these variables whenyou run any make command, causing your build andconfiguration to be screwed up.

    I Define ARCH and CROSS_COMPILE as environment variables:export ARCH=arm

    export CROSS_COMPILE=arm-linux-

    Drawback: it only works inside the current shell or terminal.You could put these settings in a file that you source everytime you start working on the project. If you only work on asingle architecture with always the same toolchain, you couldeven put these settings in your ~/.bashrc file to make thempermanent and visible from any terminal.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 111/447

  • Predefined configuration files

    I Default configuration files available, per board or per-CPUfamily

    I They are stored in arch//configs/, and are justminimal .config files

    I This is the most common way of configuring a kernel forembedded platforms

    I Run make help to find if one is available for your platform

    I To load a default configuration file, just runmake acme_defconfig

    I This will overwrite your existing .config file!

    I To create your own default configuration fileI make savedefconfig, to create a minimal configuration fileI mv defconfig arch//configs/myown_defconfig

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 112/447

  • Configuring the kernel

    I After loading a default configuration file, you can adjust theconfiguration to your needs with the normal xconfig,gconfig or menuconfig interfaces

    I You can also start the configuration from scratch withoutloading a default configuration file

    I As the architecture is different from your host architectureI Some options will be different from the native configuration

    (processor and architecture specific options, specific drivers,etc.)

    I Many options will be identical (filesystems, network protocols,architecture-independent drivers, etc.)

    I Make sure you have the support for the right CPU, the rightboard and the right device drivers.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 113/447

  • Device Tree

    I Many embedded architectures have a lot of non-discoverablehardware.

    I Depending on the architecture, such hardware is eitherdescribed using C code directly within the kernel, or using aspecial hardware description language in a Device Tree.

    I ARM, PowerPC, OpenRISC, ARC, Microblaze are examples ofarchitectures using the Device Tree.

    I A Device Tree Source, written by kernel developers, iscompiled into a binary Device Tree Blob, passed at boot timeto the kernel.

    I There is one different Device Tree for each board/platformsupported by the kernel, available inarch/arm/boot/dts/.dtb.

    I The bootloader must load both the kernel image and theDevice Tree Blob in memory before starting the kernel.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 114/447

  • Building and installing the kernel

    I Run makeI Copy the final kernel image to the target storage

    I can be uImage, zImage, vmlinux, bzImage inarch//boot

    I copying the Device Tree Blob might be necessary as well, theyare available in arch//boot/dts

    I make install is rarely used in embedded development, asthe kernel image is a single file, easy to handle

    I It is however possible to customize the make install behaviourin arch//boot/install.sh

    I make modules_install is used even in embeddeddevelopment, as it installs many modules and description files

    I make INSTALL_MOD_PATH=/ modules_installI The INSTALL_MOD_PATH variable is needed to install the

    modules in the target root filesystem instead of your host rootfilesystem.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 115/447

  • Booting with U-Boot

    I Recent versions of U-Boot can boot the zImage binary.I Older versions require a special kernel image format: uImage

    I uImage is generated from zImage using the mkimage tool. Itis done automatically by the kernel make uImage target.

    I On some ARM platforms, make uImage requires passing aLOADADDR environment variable, which indicates at whichphysical memory address the kernel will be executed.

    I In addition to the kernel image, U-Boot can also pass aDevice Tree Blob to the kernel.

    I The typical boot process is therefore:1. Load zImage or uImage at address X in memory2. Load .dtb at address Y in memory3. Start the kernel with bootz X - Y or bootm X - Y

    The - in the middle indicates no initramfs

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 116/447

  • Kernel command line

    I In addition to the compile time configuration, the kernelbehaviour can be adjusted with no recompilation using thekernel command line

    I The kernel command line is a string that defines variousarguments to the kernel

    I It is very important for system configurationI root= for the root filesystem (covered later)I console= for the destination of kernel messagesI Many more exist. The most important ones are documented in

    Documentation/kernel-parameters.txt in kernel sources.

    I This kernel command line is eitherI Passed by the bootloader. In U-Boot, the contents of the

    bootargs environment variable is automatically passed to thekernel

    I Built into the kernel, using the CONFIG_CMDLINE option.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 117/447

  • Practical lab - Compile and Boot an Android Kernel

    I Extract the kernel patchset fromAndroid Kernel

    I Compile and boot a kernel for theemulator

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 118/447

  • The Android Kernel

    Changesintroduced in theAndroid KernelFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 119/447

  • The Android Kernel

    Wakelocks

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 120/447

  • Power management basics

    I Every CPU has a few states of power consumption, frombeing almost completely off, to working at full capacity.

    I These different states are used by the Linux kernel to savepower when the system is run

    I For example, when the lid is closed on a laptop, it goes intosuspend, which is the most power conservative mode of adevice, where almost nothing but the RAM is kept awake

    I While this is a good strategy for a laptop, it is not necessarilygood for mobile devices

    I For example, you dont want your music to be turned offwhen the screen is

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 121/447

  • Wakelocks

    I Androids answer to these power management constraints iswakelocks

    I One of the most famous Android changes, because of theflame wars it spawned

    I The main idea is instead of letting the user decide when thedevices need to go to sleep, the kernel is set to suspend assoon and as often as possible.

    I In the same time, Android allows applications and kerneldrivers to voluntarily prevent the system from going tosuspend, keeping it awake (thus the name wakelock)

    I This implies to write the applications and drivers to use thewakelock API.

    I Applications do so through the abstraction provided by the APII Drivers must do it themselves, which prevents to directly

    submit them to the vanilla kernel

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 122/447

  • Wakelocks API

    I Kernel Space API#include

    void wake_lock_init(struct wakelock *lock,

    int type,

    const char *name);

    void wake_lock(struct wake_lock *lock);

    void wake_unlock(struct wake_lock *lock);

    void wake_lock_timeout(struct wake_lock *lock, long timeout);

    void wake_lock_destroy(struct wake_lock *lock);

    I User-Space API$ echo foobar > /sys/power/wake_lock

    $ echo foobar > /sys/power/wake_unlock

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 123/447

  • The Android Kernel

    Binder

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 124/447

  • Binder

    I RPC/IPC mechanism

    I Takes its roots from BeOS and the OpenBinder project, whichsome of the current Android engineers worked on

    I Adds remote object invocation capabilities to the Linux Kernel

    I One of the very basic functionalities of Android. Without it,Android cannot work.

    I Every call to the system servers go through Binder, just likeevery communication between applications, and evencommunication between the components of a singleapplication.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 125/447

  • Binder

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 126/447

  • The Android Kernel

    klogger

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 127/447

  • Logging

    I Logs are very important to debug a system, either live or aftera fault occurred

    I In a regular Linux distribution, two components are involvedin the systems logging:

    I Linux internal mechanism, accessible with the dmesgcommand and holding the output of all the calls to printk()from various parts of the kernel.

    I A syslog daemon, which handles the user space logs andusually stores them in the /var/log directory

    I From Android developers point of view, this approach hastwo flaws:

    I As the calls to syslog() go through as socket, they generateexpensive task switches

    I Every call writes to a file, which probably writes to a slowstorage device or to a storage device where writes are expensive

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 128/447

  • Logger

    I Android addresses these issues with logger, which is a kerneldriver, that uses 4 circular buffers in the kernel memory area.

    I The buffers are exposed in the /dev/log directory and youcan access them through the liblog library, which is in turn,used by the Android system and applications to write tologger, and by the logcat command to access them.

    I This allows to have an extensive level of logging across theentire AOSP

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 129/447

  • The Android Kernel

    Anonymous Shared Memory(ashmem)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 130/447

  • Shared memory mechanism in Linux

    I Shared memory is one of the standard IPC mechanismspresent in most OSes

    I Under Linux, they are usually provided by the POSIX SHMmechanism, which is part of the System V IPCs

    I ndk/docs/system/libc/SYSV-IPC.html illustrates all thelove Android developers have for these

    I The bottom line is that they are flawed by design in Linux,and lead to code leaking resources, be it maliciously or not

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 131/447

  • Ashmem

    I Ashmem is the response to these flawsI Notable differences are:

    I Reference counting so that the kernel can reclaim resourceswhich are no longer in use

    I There is also a mechanism in place to allow the kernel toshrink shared memory regions when the system is undermemory pressure.

    I The standard use of Ashmem in Android is that a processopens a shared memory region and share the obtained filedescriptor through Binder.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 132/447

  • The Android Kernel

    Alarm Timers

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 133/447

  • The alarm driver

    I Once again, the timer mechanisms available in Linux were notsufficient for the power management policy that Android wastrying to set up

    I High Resolution Timers can wake up a process, but dont firewhen the system is suspended, while the Real Time Clock canwake up the system if it is suspended, but cannot wake up aparticular process.

    I Developed the alarm timers on top of the Real Time Clockand High Resolution Timers already available in the kernel

    I These timers will be fired even if the system is suspended,waking up the device to do so

    I Obviously, to let the application do its job, when theapplication is woken up, a wakelock is grabbed

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 134/447

  • The Android Kernel

    Low Memory Killer

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 135/447

  • Low Memory Killer

    I When the system goes out of memory, Linux throws the OOMKiller to cleanup memory greedy processes

    I However, this behaviour is not predictable at all, and can killvery important components of a phone (Telephony stack,Graphic subsystem, etc) instead of low priority processes(Angry Birds)

    I The main idea is to have another process killer, that kicks inbefore the OOM Killer and takes into account the time sincethe application was last used and the priority of thecomponent for the system

    I It uses various thresholds, so that it first notifies applicationsso that they can save their state, then begins to killnon-critical background processes, and then the foregroundapplications

    I As it is run to free memory before the OOM Killer, the latterwill never be run, as the system will never run out of memory

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 136/447

  • The Android Kernel

    The ION Memory Allocator

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 137/447

  • ION 1/2

    I ION was introduced with Ice Cream Sandwich (4.0) version ofAndroid

    I Its role is to allocate memory in the system, for most of thepossible cases, and to allow different devices to share buffers,without any copy, possibly from an user space application

    I Its for example useful if you want to retrieve an image from acamera, and push it to the JPEG hardware encoder from anuser space application

    I The usual Linux memory allocators can only allocate a bufferthat is up to 512 pages wide, with a page usually being 4kiB.

    I There was previously for Android (and Linux in general) somevendor specific mechanism to allocate larger physicallycontiguous memory areas (nvmap for nVidia, CMEM for TI, etc.)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 138/447

  • ION 2/2

    I ION is here to unify the interface to allocate memory in thesystem, no matter on which SoC youre running on.

    I It uses a system of heaps, with Linux publishing the heapsavailable on a given system.

    I By default, you have three different heaps:

    system Memory virtually contiguous memory, backed byvmalloc

    system contiguous Physically contiguous memory, backed bykmalloc

    carveout Large physically contiguous memory,preallocated at boot

    I It also has a user space interface so that processes canallocate memory to work on.

    I https://lwn.net/Articles/480055/

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 139/447

  • Comparison with mainline equivalents

    I ION has entered staging since 3.14. And:I The contiguous allocation of the buffers is done through CMAI The buffer sharing between devices is made through dma-bufI Its user space API also allows to allocate and share buffers

    from the user space, which was not possible otherwise.I This API is also used to set the allocation constraints devices

    might have (for example, when one particular device can onlyaccess a subset of the memory, or when it needs to setup anIOMMU)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 140/447

  • The Android Kernel

    Network Security

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 141/447

  • Paranoid Network

    I In the standard Linux kernel, every application can opensockets and communicate over the Network

    I However, Google was willing to apply a more strict policy withregard to network access

    I Access to the network is a permission, with a per applicationgranularity

    I Filtered with the GID

    I You need it to access IP, Bluetooth, raw sockets or RFCOMM

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 142/447

  • The Android Kernel

    Various Drivers and Fixes

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 143/447

  • Various additions

    I Android also has a lot of minor features added to the Linuxkernel:

    I RAM Console, a RAM-based console that survives a reboot tohold kernel logs

    I pmem, a physically contiguous memory allocator, writtenspecifically for the Qualcomm MSM SoCs. Obsolete Now.

    I ADBI YAFFS2I Timed GPIOs

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 144/447

  • The Android Kernel

    Linux Mainline Patches Merge

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 145/447

  • History

    I The Android Kernel patches were kept for a long time out ofthe official Linux release

    I They were first integrated in 2.6.29, indrivers/staging/android

    I They were then removed from the kernel 2.6.35, becauseGoogle was unwilling to help the mainlining process

    I They were then added back in 3.3 (around 2 years later) andare still there at the time

    I While Google did a great job at keeping most of their changesas isolated from the core as possible, making this easy tomerge in the staging area, it wasnt true for the wakelocks,due to their invasive nature.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 146/447

  • Wakelocks Support

    I The kernel developpers were not quite happy about thein-kernel APIs used by the wakelocks

    I Due to the changes in every places of the kernel to statewether or not we were allowed to suspend, it was not possibleto merge the changes as is: either you were getting all of it, ornone

    I Since version 3.5, two features were included in the kernel toimplement opportunistic suspend:

    autosleep is a way to let the kernel trigger suspend orhibernate whenever there are no active wakeupsources.

    wake locks are a way to create and manipulate wakeupsources from user space. The interface iscompatible with the android one.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 147/447

  • Current State: Merged Patches

    I As of 3.10, the following patches/features are now found inthe mainline kernel:

    I BinderI Alarm Timers (under the name POSIX Alarm Timers

    introduced in 2.6.38)I AshmemI KloggerI Timed GPIOsI Low Memory KillerI RAM Console (superseded by pstore RAM backend introduced

    in 3.5)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 148/447

  • Current State: Missing Patches

    I As of 3.10, the following patches/features are missing fromthe mainline kernel:

    I Paranoid NetworkingI ION Memory AllocatorI USB GadgetI FIQ debuggerI pmem (removed in 3.3)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 149/447

  • Android Bootloaders

    AndroidBootloadersFree Electrons

    Copyright 2004-2014, Free Electrons.Creative Commons BY-SA 3.0 license.Corrections, suggestions, contributions and translations are welcome!

    Embedded LinuxDevelopers

    Free Electrons

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 150/447

  • Android Bootloaders

    Boot Sequence

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 151/447

  • Bootloaders

    I The bootloader is a piece of code responsible forI Basic hardware initializationI Loading of an application binary, usually an operating system

    kernel, from flash storage, from the network, or from anothertype of non-volatile storage.

    I Possibly decompression of the application binaryI Execution of the application

    I Besides these basic functions, most bootloaders provide a shellwith various commands implementing different operations.

    I Loading of data from storage or network, memory inspection,hardware diagnostics and testing, etc.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 152/447

  • Bootloaders on x86 (1)

    I The x86 processors are typically bundled on aboard with a non-volatile memory containing aprogram, the BIOS.

    I This program gets executed by the CPU afterreset, and is responsible for basic hardwareinitialization and loading of a small piece of codefrom non-volatile storage.

    I This piece of code is usually the first 512 bytesof a storage device

    I This piece of code is usually a 1st stagebootloader, which will load the full bootloaderitself.

    I The bootloader can then offer all its features. Ittypically understands filesystem formats so thatthe kernel file can be loaded directly from anormal filesystem.

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 153/447

  • Bootloaders on x86 (2)

    I GRUB, Grand Unified Bootloader, the most powerful one.http://www.gnu.org/software/grub/

    I Can read many filesystem formats to load the kernel image andthe configuration, provides a powerful shell with variouscommands, can load kernel images over the network, etc.

    I See our dedicated presentation for details:http://free-electrons.com/docs/grub/

    I Syslinux, for network and removable media booting (USB key,CD-ROM)http://www.kernel.org/pub/linux/utils/boot/syslinux/

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 154/447

  • Booting on embedded CPUs: case 1

    I When powered, the CPU starts executing codeat a fixed address

    I There is no other booting mechanism providedby the CPU

    I The hardware design must ensure that a NORflash chip is wired so that it is accessible at theaddress at which the CPU starts executinginstructions

    I The first stage bootloader must be programmedat this address in the NOR

    I NOR is mandatory, because it allows randomaccess, which NAND doesnt allow

    I Not very common anymore (unpractical, andrequires NOR flash)

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 155/447

  • Booting on embedded CPUs: case 2

    I The CPU has an integrated boot code in ROMI BootROM on AT91 CPUs, ROM code on OMAP, etc.I Exact details are CPU-dependent

    I This boot code is able to load a first stage bootloader from astorage device into an internal SRAM (DRAM not initializedyet)

    I Storage device can typically be: MMC, NAND, SPI flash,UART, etc.

    I The first stage bootloader isI Limited in size due to hardware constraints (SRAM size)I Provided either by the CPU vendor or through community

    projects

    I This first stage bootloader must initialize DRAM and otherhardware devices and load a second stage bootloader intoRAM

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 156/447

  • Booting on ARM Atmel AT91

    I RomBoot: tries to find a valid bootstrap imagefrom various storage sources, and load it intoSRAM (DRAM not initialized yet). Size limitedto 4 KB. No user interaction possible in standardboot mode.

    I AT91Bootstrap: runs from SRAM. Initializes theDRAM, the NAND or SPI controller, and loadsthe secondary bootloader into RAM and starts it.No user interaction possible.

    I U-Boot: runs from RAM. Initializes some otherhardware devices (network, USB, etc.). Loads thekernel image from storage or network to RAMand starts it. Shell with commands provided.

    I Linux Kernel: runs from RAM. Takes over thesystem completely (bootloaders no longer exists).

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://free-electrons.com 157/447

  • Booting on ARM TI OMAP3

    I ROM Code: tries to find a valid bootstrap imagefrom various storage sources, and load it intoSRAM or RAM (RAM can be initialized by ROMcode through a configuration header). Sizelimited to

  • Booting on Marvell SoC

    I ROM Code: tries to find a valid bootstrap imagefrom various storage sources, and load it intoRAM. The RAM configuration is described in aCPU-specific header, prepended to the bootloaderimage.

    I U-Boot: runs from RAM. Initializes some otherhardware devices (network, USB, etc.). Loads thekernel image from storage or network to RAMand starts it. Shell with commands provided. Filecalled u-boot.kwb.

    I Linux Kernel: runs from RAM. Takes over thesystem completely (bootloaders no longer exists).

    Free Electrons. Embedded Linux, kernel, drivers and Android development, consulting, training and support. http://fr