Advanced Programmable Interrupt Controller

8/8/2019 Advanced Programmable Interrupt Controller

1/28

Programmable Interrupt ControllerThough I had originally wrote my APIC codeover a year ago, I've been playing with itrecently, so I thoughtI'd write about it. This is not a completetreatment, but it contains stuff that the docs forthe chips don't seemto tell.There are basically two things here to consider.

Built into all recent x86 CPU chips (Pent Proand up) is a thing called a Local APIC. It isaddressed atphysical addresses FEE00xxx. Actually, that isthe default, it can be moved by programmingthe MSR

that holds it base address.It has many fun things in it. The big thing is thatyou can interrupt other CPU's in amultiprocessorsystem. But if you just have a uniprocessor,there are useful things for it, too.

.1.


2/28

Some motherboards have an IO APIC on them.This is usually only found on multiprocessorboards.Functionally, it replaces the 8259's. You mustessentially shut off the 8259's and turn on theIO APIC touse it.The IO APIC is typically located at physicaladdress FEC00000, but may be moved by

programmingthe north/southbridge chipset.The Intel chip number is 82093 and you can getthe doc for it off of the Intel website.2.What the Local APIC Is

As stated above, the Local APIC (LAPIC) is acircuit that is part of the CPU chip. It containsthese basicelements:A mechanism for generating 1. interrupts2. A mechanism for accepting interrupts

3. A timerIf you have a multiprocessor system, the APIC'sare wired together so they can communicate.So the LAPIC


3/28

on CPU 0 can communicate with the LAPIC onCPU 1, etc.What the IO APIC Is

This is a separate chip that is wired to the LocalAPIC's so it can forward interrupts on to theCPU chips. It isprogrammed similar to the 8259's but has moreflexibility.It is wired to the same bus as the Local APIC's

so it can communicate with them.Fun things to do with a Local APIC in aUniprocessorAdvanced Programmable Interrupt Controller2 of 8(this stuff also applies to multiprocessors,

too)One thing the LAPIC can help with is thefollowing problem:An IRQ-type interrupt routine wishes to wake asleeping thread, but this IRQ interrupt may benested several

levels inside other IRQ interrupts, so it cannotsimply switch stacks as those outer interruptroutines wouldnot complete until the old thread is re-woken.


4/28

So we have to somehow switch out of thecurrent thread and switch into the thread to bewoken. A way theLAPIC can help us is to tell it to interrupt thissame CPU, but only when there are no IRQ-type interrupthandlers active.I call this a 'software' interrupt because theoperating system software initiated the

interrupt. It is programmedinto the LAPIC to be at a priority lower than anyIRQ-type interrupt.So now if some IRQ-type routine wants to wakea thread, it makes the necessary changes tothe

datastructures, then triggers a softwareinterrupt to itself. Then, when all IRQ-typeinterrupt handlers havereturned out, the LAPIC is now able tointerrupt.It interrupts out of the currentlyexecuting thread and

switches to the thread that was just woken.Very neat.


5/28

Without the LAPIC, your interrupt routine has toset a flag in memory somewhere that eachIRET has tocheck for. So each IRET checks this flag andchecks to see if it is the 'last' IRET. It is moreefficient to let theLAPIC do this testing for you.So now we have to make this software LAPICinterrupt have a lower priority than IRQ

interrupts.We do thisby studying how the LAPIC assigns priority tointerrupts. This is a bit lame but it works ok. Thepriority isbased on the vector number we choose for theinterrupt. Interrupt vectors are numbered 0x00

through 0xFF inIntel CPUs. The LAPIC assigns a priority basedon the first of the two hex digits and ignores thesecond digit.Thus, any interrupts using vectors 0x50 through0x5F have the same priority. So if you block

something atpriority 0x52, you block all interrupts in therange 0x50 through 0x5F.


6/28

Now the CPU itself uses vectors in the range0x00..0x1F for exceptions, so we don't want touse those forLAPIC interrupts. This means we can use avector numbered 0x20 or 0x2F or somewherein that range.Wewill have to redirect the IRQ interrupts tovectors 0x30..0x3F or something even higher ifnecessary, by

re-programming the 8295's. Now we can blocksoftware interrupts without blocking IRQinterrupts.The LAPIC's priority can be set by writing theLAPIC's TSKPRI (task priority) register. So ifyou want to

block all interrupts through level 0x2F, just writea 0x20 (or 0x2B, etc) into the TSKPRI and youhaveblocked those interrupts.Now the LAPIC is not really connected to the8259's. You cannot block 8259 generated

interrupts with theLAPIC. Likewise, being in an IRQ-type interrupthandler does not block any LAPIC interrupts.So we have to


7/28

manually block/unblock the softints at thebeginning of our IRQ handler. Just push theLAPIC's TSKPRIregister, set it to 0x20 and handle your IRQinterrupt as usual.When done, pop the savedLAPIC's TSKPRIthen IRET.So the IRQ interrupt handler looks somethinglike:

entry: # cpu hw ints inhibited on entrypush LAPIC's TSKPRI # save previousTSKPRI, either 00 or 20movl $0x20, LAPIC's TSKPRI # make suresoftint delivery inhibited nowsti # let cpu process higher priority interrupts

push general registers # save registers used byinterrupt handlersAdvanced Programmable Interrupt Controller3 of 8process interrupt # process the IRQ interruptlevel

pop general registers # restore clobberedregisterscli # prevent nesting until we have unwoundstack


8/28

send EOI to the 8259's # let 8259's deliver thisIRQ level again after iretpop LAPIC's TSKPRI # restore TSKPRI,possibly allowing softints after iretiretThe software interrupt handler looks somethinglike:entry: # cpu hardware interrupt delivery enabledmovl $0x20,LAPIC's TSKPRI # prevent nesting

of software interruptssend EOI to LAPIC # allow local APIC to queueanother interruptsave general registersswitch stack to highest priority threadrestore general registers

cli # prevent nesting until we have unwoundstackmovl $0,LAPIC's TSKPRI # allow local APIC todeliver softints after the iretiretSomething else the Local APIC is good for

in a Uniprocessor:Another use for the LAPIC is that it has a builtin timer. So you can set the timer for anyinterval and it will


9/28

generate an interrupt.It basically has a 32-bitcounter that runs at the bus speed, like100MHz. So what I use itfor is the quantum reschedule interrupt. If asingle thread uses the CPU for a whole tenth ofa second, thistimer interrupts it and lowers the thread'spriority. Then if there is an equal priority threadwaiting, it switches

to it.This is particularly attractive in a multiprocessorsystem, as there is one such timer per CPUchip, so you haveone quantum timer per executing thread to workwith.

The quantum timer interrupt handler is verysimilar to the regular softint handler, except itdecrements thecurrent thread's priority. So I put the quantumtimer on a vector in the 0x20..0x2F range rightnext to the

regular softint vector. Thus writing 0x20 to theTSKPRI register blocks both softints andquantum ints.


10/28

Now I couldn't resist the temptation to muck itup by folding in my normal timer queue stuff asI can getbasically 10nS resolution on my timer requests.So I have it designed such that whichever CPUqueued thenext-to-expire timer request, gets its LAPICtaken over to service the timer queue, untilanother CPU comes

along with an even earlier timer request, or thethread's quantum is going to run out earlier thanthe timerrequest. So my current timer handler is actuallya bit more complicated than the one shownabove.

Multiprocessor uses for the Local APICHaving the Local APIC is essential in amultiprocessor.Without it, the motherboardwould have to providesome equivalent functionality.One function fits in nicely with the 'softint' stuff

described for the uniprocessor.We justgeneralize it a bit. In


11/28

a general multi-processor OS, an IRQ interrupthandler just needs to 'wack' *some* CPU tohandle the newlywoken thread, not necessarily its own. With theLAPIC, all it has to do is tell its LAPIC to send amessage tothe current lowest priority LAPIC that it isconnected to, possibly including itself.Something unique to multiprocessor systems is

the following situation:Two or more processors are mapped to thesame process at the same time, presumablyrunningdifferent threads in that process1.

Advanced Programmable Interrupt Controller4 of 8One of the CPU's (threads) unmaps a memoryregion and marks the corresponding pagetableentries asbeing 'not-present'

2.The CPU executes INVLPG instructions to wipeout any left over cached copies of the pagetableentries it has


12/28

3.Now that CPU has to tell the other CPU toinvalidate any cached pagetable entries it has.It can do thiswith the LAPIC's. It sends an interrupt out to tellthe other CPU's to invalidate their pagetableentries atthat virtual address.4.

I use a very high priority interrupt for thispurpose, even higher than any IRQ interrupts.This is because Ihave the originating CPU wait for the targetCPU's to acknowledge that they haveprocessed the invalidate

before continuing on. So I want the targetCPU's to process the interrupt immediately.Another fun thing to do with LAPIC's in amultiprocessor:You have implemented a kernel debugger. OneCPU hits a breakpoint and traps to the kernel

debugger. Theother CPU's are still merrily churning awaydoing who-knows-what. So you can interruptthem (my OS sends


13/28

them an NMI) so they will also call thedebugger. This way you will have stabledatastructures to examineand you will also be able to see just what theother CPU's were doing at the time of thebreakpoint.What the IOAPIC is good forThe IOAPIC is generally only used inmultiprocessor systems, and is not typically

found on uniprocessormotherboards.The main advantage it has over 8259's is that itcan distribute the IRQ type interrupts to variousCPU's. The8259's will only deliver the interrupt to the boot

CPU, whilst ignoring the others.Another small advantage it has is that it canroute the PCI interrupts to separate vectors andnot overlap theIRQ's. My OS does not take advantage of this,but you may want to take on this challenge.

The IOAPIC has a table of 24 64-bit registers.This was extreme overkill, IMO, as less than 32bits are used


14/28

from each. But anyway, that's the way it is.Also, the registers are accessed indirectly,there is an 'address'register and a 'data' register. Fortunately, likethe 8259 and unlike the LAPIC, the IOAPIC canbeprogrammed at boot time then left alone for therest of system operation.Each of the 24 registers has an interrupt source

that is hardwired from the motherboard design.Standarddesigns follow this convention:IRQ's 0..15 : IOAPIC registers 0..15PCI A..D : IOAPIC registers 16..19I do not know what they use 20..23 for. It is

possible that 23 is used for SMI, but I haven'tever needed to useit, so I haven't research it. This is left as anexcercise to the reader.The trick to programming these things is settingthe polarity and trigger mode. For the IRQ 0..15

interrupts,set it to 'edge triggered, active high'mode (both and zero). For the PCI A..Dinterrupts, set it to


15/28

'level triggered, active low' (both and one). At least that's what works on mymotherboard.Now here's where you can get creative. Each ofthe interrupts can be programmed with any ofthe 224 vectorsthe CPU is capable of (we don't use0x00..0x1F, just 0x20..0xFF). Like the LAPIC's,the IOAPIC's will

assign a priority to the interrupt based on thevector you give it. And also like the LAPIC's, thedumb thingsAdvanced Programmable Interrupt Controller5 of 8only use the 4 top bits of the vector number to

distinguish priority, and ignore the low 4 bits. Sowe have tocompromise and group 2 interrupts per level.So you want to assign the lowest priorityinterrupts to the lowestvector numbers.

Conventionally, from highest to lowest priority,the IRQ's go0,1,2,8,9,10,11,12,13,14,15,3,4,5,6,7. You can


16/28

assign the vectors to the IRQ's in that order orany order you wish.Whichever IRQ you givethe highestvector number to will be the highest priority, etc.I just stuck with the conventional assignment asI can't thinkof any reason not to.So we have 20 vectors to assign, 16 for theIRQ's and 4 for the PCI's. As I said, in my OS I

map the PCIinterrupts onto the IRQ's like the BIOS set up,you may want to do different. So all I reallyneed is 16vectors.I want to leave some at the bottom (forsoftint stuff) and some at the top (for high-

priority LAPICinterrupts).So I can get away with using vector numbers0x74, 0x7C, 0x84, 0x8C, ... 0xE4, 0xEC. Iassign these to theusual IRQ priorities 7 (lowest), 6, 5, 4, ..., 1, 0.

This means I program IOAPIC interrupt register0 with the


17/28

vector for IRQ 0 which I want to be 0xEC, asIRQ 0 is the highest priority. I program IOAPICinterruptregister 1 with 0xE4 as IRQ 1 is the nexthighest priority. And so on, until I haveprogrammed the first 16 ofthe IOAPIC's interrupt registers.I seem to get useless interrupts on the IRQ 2line, so I set this table entry to be disabled.Now for the truly exciting part, programming thenext four interrupt registers that correspond toPCIinterrupts A..D. I chickened out here. It wouldbe nice to program these completelyindependent of the IRQ

interrupts. Problem is, almost all controllercards report that they use INT-A to send theirinterrupt. So themotherboards scramble the actual vectorsaround, so INT-A on slot 10 might actuallycome in as INT-B to the

IOAPIC, and INT-A on slot 9 might come in asINT-C to the IOAPIC. Only the BIOS (or mobotech doc)knows the answer to this puzzle.


18/28

It would have been nice if the PCI BIOS wouldhave provided a call to retrieve this nightmarebut it doesn't.So what I do is go out to thenorthbridge/southbridge chipset and retrieve thePCI-to-IRQ mapping that theBIOS has established. In the PIIX4 chipset, thiscan be retrieved from the longword at offset0x60 of device

id 0x7110 (vendor id 0x8086). Byte 0 of thislong tells me what IRQ it is going to forwardPCI-A interruptsto, so I program the IOAPIC to do the samething by setting the IOAPIC interrupt register 16to interrupt to

the same vector used by the IRQ. So if byte 0has a 5, it means it has set up controllersconnected to PCI-Ainterrupts to think they are IRQ 5's, so I putvector 0x84 in the IOAPIC's interrupt register16. Likewise, byte

1 of the long tells me what IRQ it has set INT-Bcards to, so I write the vector number I setaside for that IRQ


19/28

for the IOAPIC's interrupt register 17. Same forbytes 2 and 3. If the BIOS didn't find any PCIcards using agiven PCI interrupt line, it sets the byte to 0x80indicating that it is disabled.Finally, I simply disable entries 20 through 23.So what I end up with is:IOAPIC int reg Vector0 0xEC

1 0xE42 disabled3 0x944 0x8C5 0x846 0x7C

7 0x748 0xD49 0xCCAdvanced Programmable Interrupt Controller6 of 810 0xC4

11 0xBC12 0xB413 0xAC14 0xA4


20/28

15 0x9C16 depends on what BIOS set up for PCI-A.For example if 5, use same as for IRQ 5, 0x8417 depends on what BIOS set up for PCI-B.For example if 9, use same as for IRQ 9, 0xCC18 depends on what BIOS set up for PCI-C.For example if 10, use same as for IRQ 10,0xC419 depends on what BIOS set up for PCI-D.

For example if 12, use same as for IRQ 11,0xBC20..23 disabledNow if that doesn't make your head hurt, notonly can you tell the IOAPIC what vector to useto process the

interrupt, you can tell it which CPU to send theinterrupt to. Here, I take the easy way out. I tellit to send it tothe CPU that is currently at the lowest priority.By priority, we mean the higher of what's in itsLAPIC

TSKPRI register or whatever interrupt it iscurrently servicing.


21/28

One more little thing. Since these interruptregisters are 64 bits, it takes two writes to setthem. So what I dois disable the entry by writing a disable value(0x10000) to the low half, then I write thedesination info to thetop half, then finally set the low half to what Iwant.Oh did I say one more thing? There's more.

Some motherboards have a thingy that youmust write to redirectthe interrupts from the 8259's to the IOAPIC.This is done by writing 0x70 to port 0x22 and0x01 to port0x23. Also, you may want to disable the 8259's

completely. I do this by writing 0xFF to ports0x21 and 0xA1.So I disable the 8259's first, then program theIOAPIC, then write the ICMR (0x22,0x23) portsto acceptIOAPIC interrupts.

From this point on, the IOAPIC is programmedand you shouldn't have to access it again.


22/28

When the IOAPIC receives an interrupt from acontroller card, it forwards the interrupt out tothe LAPIC's (asthey are all wired together).When an LAPIC isable to accept the interrupt (ie, its TSKPRI islower than thevector and it is not currently servicing the sameor higher level, and the CPU itself has interruptdelivery

enabled), the LAPIC will signal an interrupt tothe CPU and the CPU will interrupt through thecorrespondinginterrupt vector that was programmed into theIOAPIC.Using the programming table above, to inhibit

delivery of IRQ 7 and below in a particular CPU,just write0x8C to the LAPIC's TSKPRI register. Thisblocks delivery of all APIC generated interruptsin that CPU withvectors numbered up through 0x8F. Other

CPU's are free to accept those interrupts,though (assuming their


23/28

corresponding TSKPRI register is not blockingthem). So if you want to block interrupts on ALLCPU's for agiven level, you will have to set a spinlock aftersetting the TSKPRI register. And your interruptroutine willalso have to lock that same spinlock. So ifanother CPU gets that IRQ 7 interrupt while youhave the spinlock,

it will wait for you to release the spinlock beforecontinuing.While not truly blocking the interrupton otherCPU's, it has the same basic effect.MiscPhysical/logical id/destination

When an multiprocessor system starts, thehardware loads a physical id number into eachCPU's LAPICcircuit. This is how you can tell one CPU fromanother in the system. You can also reprogramthis number if

Advanced Programmable Interrupt Controller7 of 8


24/28

you wish, but generally there's no need to. It'ssufficient that each LAPIC have a uniquephysical id.So if you set an IO or Local APIC interruptregister to target a particular physical id, then itwill go to thatCPU. This may be fine for some instances, butthere are cases where we want to interrupt agiven set of

CPU's.Well we could send an interrupt to eachone individually. But that is inefficient and thereis a betterway. Enter the 'logical destination'. This is abitmask thing. If a sender has a bit set in itstarget mask that a

destination has in its logical address mask, thattarget is eligible to receive the interrupt. Sowhat I do is setthe following logical addresses up:physical id 0 gets logical id 0x01physical id 1 gets logical id 0x02

physical id 2 gets logical id 0x04physical id 3 gets logical id 0x08...physical id 7 gets logical id 0x80


25/28

This scheme limits me to 8 CPU's (as thelogical destination mask is only 8 bits), but ifyou have more CPU'sthen you can double them up (eg, 0&8 getmask 0x01, 2&9 get mask 0x02, etc). TheAPIC's also support aclustering mode, but I had no need for it so Ididn't study it.Assuming we have less than 9 CPU's, we can

easily select the targets by setting theappropriate bits in ourinterrupt mask. Like for the IOAPICdestinations, I set all the bits so any CPU's Ihave are elegible for theinterrupt, and I tell the IOAPIC to select the

lowest priority amongst all those CPU's.When Ihave toinvalidate a shared pagetable entry, I can selectonly those CPU's that are also currently usingthat samepagetable and leave the others alone.

Local APIC initializationThe Local APIC contains stuff that has to beinitialzed. It may be that the BIOS sets it for theboot processor


26/28

so it will behave normally, but you will need toinitialize it for any other processors, so youmight as well doit for all so they come out the same.Here's what I put in it:task priority (FEE00080) = 0x20 to inhibit softintdeliverytimer interrupt vector (FEE00320) = 0x10000 todisable timer interrupts

performance counter interrupt (FEE00340) =0x10000 to disable performance counterinterruptslocal interrupt 0 (FEE00350) = 0x08700 toenable normal external interruptslocal interrupt 1 (FEE00360) = 0x00400 to

enable normal NMI processingerror interrupt (FEE00370) = 0x10000 to disableerror interruptsspurious interrupt (FEE000F0) = 0x0010F toenable the APIC and set spurious vector to 15Then after it is enabled, I set the local interrupt

0 and 1 again to the same thing, just to be surethe reset didn'tkeep them shut off.


27/28

That's another thing the book doesn't quite say.Local interrupt 0 is for interrupts on the CPU'snormalinterrupt pin. Local interrupt 1 is for interruptson the CPU's NMI pin. Maybe it's in theresomewhere, but Ithought I state it here in case it's not or youcan't find it.Finally, I program the logical destination

address from the physical id, puquing (fr) if thephysical id is .ge. 8.Advanced Programmable Interrupt Controller8 of 8A little gotchaTo tell the LAPIC to generate an interrupt, you

have to write two registers. In my OS (andprobably all thatuse the thing), interrupt routines are allowed towrite the LAPIC registers. So you have todisable the CPU'sinterrupts while writing the two LAPIC registers

or you will have a mess:pushflclimovl target_mask, LAPIC's ICREG 1


28/28

Advanced Programmable Interrupt Controller

Documents