More about Linaro Connect: connect.linaro.org More about Linaro: www.linaro.org/about/ More about Linaro engineering: www.linaro.org/engineering/ Hong-Kong (LCE13)
LCA13: Who Disturbs My Slumber
May 26, 2015
Resource: LCA13
Name: Who Disturbs My Slumber
Date: 05-03-2013
Speaker: Daniel Lezcano
Name: Who Disturbs My Slumber
Date: 05-03-2013
Speaker: Daniel Lezcano
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
More about Linaro Connect: connect.linaro.org More about Linaro: www.linaro.org/about/
More about Linaro engineering: www.linaro.org/engineering/
Hong-Kong (LCE13)
2EUROPE 2012 (LCE12)
www.linaro.org
Linaro Connect Hong-Kong 2013
Who disturbs my slumber ?!
3EUROPE 2012 (LCE12)
www.linaro.org
Introduction
The power management is a wide area
Power aware scheduler, P-States, C-States, optimizations for idling the cpu, regulators ...
We will focus on the sources of wake up for ARM architecture
5EUROPE 2012 (LCE12)
www.linaro.org
Introduction
“Idle” opposite to “Running”, in between an event occurs.
We want a cpu to be idle as much as possible without impacting the performance of the system
The cpuidle framework takes care of entering the idle state depending on the predictable events on the system
The depth of the sleep will depend on the next event on the system
What is this event ? An interrupt.
6EUROPE 2012 (LCE12)
www.linaro.org
Idle vs Running
When there is no more tasks to run, the scheduler choose the special idle task
This one is an infinite loop entering and exiting the arch specific idle function
When an interrupt occurs, the function exits and the idle task is yield or enters again into the idle function
9EUROPE 2012 (LCE12)
www.linaro.org
Package Idle
The deep idle state could be reached only when all cpus are idle
One cpu exiting the idle state will lead the package to exit the idle state also
Software idle states management is challenging on multicore SoCs
11EUROPE 2012 (LCE12)
www.linaro.org
/proc/interrupts – Snowball U8500
CPU0 CPU1 29: 3150 4124 GIC twd 36: 783 0 GIC Nomadik Timer Tick ... 57: 1592 0 GIC dma40 58: 73 0 GIC uart-pl011 72: 0 0 GIC ab8500 79: 187 0 GIC prcmu 87: 0 0 GIC nmk-i2c 92: 4687 0 GIC mmci-pl18x (cmd)131: 14897 0 GIC mmci-pl18x (cmd)...IPI0: 0 0 CPU wakeup interruptsIPI1: 0 167 Timer broadcast interruptsIPI2: 386 4908 Rescheduling interruptsIPI3: 0 0 Function call interruptsIPI4: 3 67 Single function call interruptsIPI5: 0 0 CPU stop interruptsErr: 0
12EUROPE 2012 (LCE12)
www.linaro.org
/proc/interrupts – Pandaboard OMAP4
CPU0 CPU1 29: 293 395 GIC twd 41: 0 0 GIC l3-dbg-irq 42: 0 0 GIC l3-app-irq 44: 0 0 GIC DMA 69: 61 0 GIC gp_timer 88: 0 0 GIC i2c.9 89: 0 0 GIC i2c.10 93: 0 0 GIC i2c.11 94: 0 0 GIC i2c.12106: 93 0 GIC OMAP UART2169: 0 0 PRCM hwmod_ioIPI0: 0 0 Timer broadcast interruptsIPI1: 1424 1260 Rescheduling interruptsIPI2: 0 0 Function call interruptsIPI3: 81 90 Single function call interruptsIPI4: 0 0 CPU stop interruptsErr: 0
13EUROPE 2012 (LCE12)
www.linaro.org
Challenging the interrupts
Where are they coming from ?
When do they occur ?
How can we reduce them ?
14EUROPE 2012 (LCE12)
www.linaro.org
From where ?
Hardware interrupt
Network, keyboard, mouse, timer, MMC, USB, serial, ...
Inter Processor Interrupt : Wisely used today, they are tricky to optimize
The architecture differs across the SoC vendor but they are slightly similar for the IPI
15EUROPE 2012 (LCE12)
www.linaro.org
Hardware interrupt
HW interrupt can happen at arbitrary times
The system could be tweaked depending on the hardware
One deterministic interrupt is interesting : the timer
16EUROPE 2012 (LCE12)
www.linaro.org
Hardware interrupt
Interrupt Deterministic Comment
Network No The network stack switch to polling on high traffic
Keyboard No
Mouse No
MMC No
Timer Yes Two kinds of timer
USB No
Serial No
17EUROPE 2012 (LCE12)
www.linaro.org
Timers
A complex infrastructure to handle most of the kernel and time services for userspace programs
Two kinds of timer
Per cpu timer
Global timer
18EUROPE 2012 (LCE12)
www.linaro.org
Timers
Timer Watchdog (twd)The timer is local to the CPUAlso known as “local timer”It goes down when the processor logic is shutdown
Timer device (architecture dependant)Less accurate (eg. 32KHz)Usually always out of the CPU’s power domainUsed as backup when the CPU are sleeping
19EUROPE 2012 (LCE12)
www.linaro.org
Timers
Used by the kernel for:Network stack, especially the TCP/IP protocolTimed-out IODelayed work queuesScheduling tasks...
Used by the userspace for:Asynchronous IO and timeoutThreading timed lockMainloop (poll, epoll, select)Posix timers...
In other words : widely used in the system
20EUROPE 2012 (LCE12)
www.linaro.org
IPI : Inter Processor Interrupt
A softirq
Limited to 16 on ARM with the GIC
5 used nowadaysIPI0 : defined but unused so farIPI1 : timer broadcastIPI2 : Rescheduling interruptIPI3 : Function Call interruptIPI4 : Single Function Call interruptIPI5 : Cpu Stop interrupt
21EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (1/3)
Occurs only if SMP and cpuidle with at least retention mode state
Broadcast interrupt could occur on any cpu:idle or notconcerned by the timer expiration or not
Could be optimized with dynamic timer irq affinityhttps://lkml.org/lkml/2013/2/19/555
A summary of how timer broadcast workshttps://lkml.org/lkml/2013/2/20/216
22EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (2/3)
[1] : the cpuidle driver tells the time framework the local timer is no longer a valid source
[2] : the timer framework changes the source and program the timer device
[3] / [3'] : the cpuidle driver power downs the cpus
23EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (3/3)
[1] : the timer expires raising an interrupt
[2] : the cpu is woken up
[3] : the cpu handles the timer callback and through the time framework …
[4] … it sends an IPI to the cpu which are concerned by the expiration of this timer
24EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (1/4)
Used by the scheduler to wake up a processor in order to run a task
Could happen for different reasons:
Load balancing / process migration
An event occurs for a specific taskIO (eg. network ingress packet)A lock has been released for a blocked taskSignal delivery
25EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (2/4)
CPU0 is running an application
CPU1 is idle
A task on CPU1 is blocked on a lock
26EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (3/4)
The application on CPU0 releases the lock
The scheduler takes the decision to wake up the CPU1...
… and send an IPI to it
27EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (4/4)
The CPU1 is awake
The scheduler put in the run queue the task which was previously blocked
28EUROPE 2012 (LCE12)
www.linaro.org
IPI3 : Function Call interrupt
Used to do a remote function invocation on all the CPUs when the code must be run in the processor context
Setting the timer frequency
Setting up the clock event notify framework
It happens rarely on the ARM system, but often on x86 system under the term of “TLB shootdowns” which happens at fork time
29EUROPE 2012 (LCE12)
www.linaro.org
IPI3 : Function Call interrupt
Used to do a remote function invocation on all the CPUs when the code must be run in the processor context
Setting the timer frequency
Setting up the clock event notify framework
It happens rarely on the ARM system, but often on x86 system under the term of “TLB shootdowns” which happens at fork time
30EUROPE 2012 (LCE12)
www.linaro.org
IPI4 : Single Function Call interrupt
Same as the IPI3 but for a targeted CPU
Happens rarely on the system
31EUROPE 2012 (LCE12)
www.linaro.org
How to reduce the number of interrupts ?
Understand the framework and identify the unnecessary wake up
Deferrable timershttp://lwn.net/Articles/228143/
Round jiffieshttp://lkml.org/lkml/2006/10/10/189
RCU no callbackhttp://lwn.net/Articles/522262
Timer and workqueue migration on non idle cpuhttps://lkml.org/lkml/2012/9/27/188
32EUROPE 2012 (LCE12)
www.linaro.org
Deferrable timers
Used for non critical timers and change their expiration to occur when the CPU wakes up
How to use it ?
Identify non critical timers
Flag them deferrable
They will be handled when the CPU wakes up for another reason
33EUROPE 2012 (LCE12)
www.linaro.org
Round jiffies
Used to group timers to expire at the same time slot
How to use it ?
Identify timers for non precise timeout
Group them to the same time slot by using round_jiffies
34EUROPE 2012 (LCE12)
www.linaro.org
Timer and workqueue migration
Schedule work onto running CPUs instead of waking up an idle one
Still work in progress
35EUROPE 2012 (LCE12)
www.linaro.org
Misc improvements
The kernel does not do everything, userspace applications have to be improved for power management
The application could group the timers to expire at the same time if high precision is not needed
Round-jiffies like framework ?
A specific flag for timer syscalls, in order to make them deferrable ?
prctl : PR_SET_TIMERSLACK
36EUROPE 2012 (LCE12)
www.linaro.org
Misc improvements
Kill all busyloop and replace them with an event based mainloop
timerfd, signalfd, eventfd, etc …
Increase MTU size for private network area when possible
Use CPU affinity when multiple tasks have to run to sequentially
Put pressure on developers to fix the applications and make them more power aware, the kernel can not overcome bad applications
Related Documents
