Ksplice - Keep your Database systems up to date with no downtime

Oracle Linux KspliceKEEP YOUR DATABASE SYSTEMS UP TO DATE WITH NO DOWNTIME

Luis & Yves

▪ Luís Marques

▪ Oracle ACE

▪ Lisbon

▪ http://lcmarques.com / @drune / [email protected]

▪ Yves Colin

▪ Oracle ACE Associate

▪ Paris

▪ http://ycolin.wordpress.com / @ycolin / [email protected]

ksplice - A piece of history (before Oracle)

▪ Created by Jeffrey Arnold, Tim Abbott, Waseem Daher, and Anders Kaseorg at MIT

▪ Released as open source (GPL v2 license) - Latest open source version: 0.9.9

▪ Ksplice (as company) started in 2008 and got 700 customers

▪ Support for RHEL, CentOS, Fedora, Debian, Ubuntu, CloudLinux and Scientific Linux

▪ Announced in LKML as “A system for rebootless kernel security updates” – (23 April 2008) (http://lwn.net/Articles/279378/)

ksplice - A piece of history (after Oracle)

▪ Oracle acquired Ksplice in 2011.

▪

▪ Offers feature only to Oracle Linux Premier Support customers

▪ Support for Oracle Linux (UEK, RH compatible kernels) via ULN – (also supports RHEL as legacy customers(?))

▪ Support for Ubuntu and Fedora “Desktop” versions as “free community protection”

▪ Strategic move for Oracle: mission critical systems run Linux (eg: Databases, Exadata, ODA)

ksplice – Why use it?

Allows to install security and reliability patches without reboot and without downtime

Downtime (and other) dilemmas:

▪ Huge security Linux kernel hole discovered and few Oracle single instance without DR?

▪ Factory Control Systems, Energy Management System Critical Applications

▪ Rollback an update without another emergency reboot?

▪ SLA or Security?

ksplice – Installation

Kernel Patching – Manual (Recommended)

# wget -N https://www.ksplice.com/uptrack/install-uptrack

# sh install-uptrack ACCESS_KEY

# uptrack-upgrade –y

Kernel Patching - Automatic Mode

# sh install-uptrack ACCESS_KEY –autoinstall

or edit /etc/uptrack/uptrack.conf and change autoinstall = yes

ksplice – Your first live demo with ksplice

Install and view all kernel updates available

# uptrack-show –-available # uptrack-show

# uptrack-install <ksplice_id> -y

# uptrack-upgrade -y

# uptrack-uname –a

ksplice – Your first live demo with ksplice

You can patch your system without any reboot but remove them rebootless as well

# uptrack-remove <ksplice_id>

# uptrack-remove –all

# uptrack-uname –a

Exploit, not exploit

ksplice and Oracle Database

▪ Ksplice doesn’t patch any Oracle shared libraries or binaries

▪ You want to patch your Database that is running of top of ext4 (?)

▪ Oracle instrumentation doesn’t catch it (?)

▪ Nothing in “Wait event” interface

▪ A few more cpu cycle are recorded during and after the patch, but is negligible.

ext4_file_open needs to be

patched but it is in use by

Oracle and your OS

The thread that can run the

function is suspended for

<1ms – No CPU can execute

it

Execution is resumed<1ms

Ksplice ,Exadata and AWS

▪ Email from Ksplice support about Exadata:

▪ Check against OVM on EXADATA version 12.1.2.2.1.160119

▪ Tested under Redhat 7.3 EC2 image

▪ Possible on AWS AMI customized XEN kernel (“Supported for legacy customers”)

Ksplice – Under the wood

Kplice – How patch is created? (1/2)

Source Code of Running Kernel

Slice

Source Code of Patch

Ksplice Build Environment:

• Pre Object Code

• Post Object Code Ksplice-id.koKsplice-id-vmlinux-

new.ko

Ksplice-id-vmlinux-

old.ko

Identify modified functions

Object Level Code difference

Patch Created

Slice

Kplice – How patch is created? (2/2)

▪ Ksplice compares code at object code level

▪ In this way Ksplice patch creation could be completely automated(in most of the cases)

▪ Ksplice compiles code(patched/non patched) with special gcc options -ffunction-sections -fdata-sections

Patch creation magic (1/2)

Kplice – How patch is applied?

Connect to the

repository

Download the

Splice (tar.gz)

Files in

/var/cache/uptrack/

Linux/$arch/$kernel

Ksplice-id.ko core module

interface

(userland – kernel land)

Ksplice-id-vmlinux-new.ko Ksplice-id-vmlinux-old.ko

Interaction via

/sys/modules/ksplice-id/ksplice

Inspect thread – patched functions

in execution? (safety check)

Execute stop_machine() to

suspend thread

Patched code loaded in

memory

Patch applied

Thread

execution

resumed

Kplice – How patch is applied (in deep) ?

Compare Running Code in

memory to pre-code(safety check)

Resolve Undefined Symbol in Post

Code/Patched code (where are

they located in running kernel?) pre

matching phase

Find the patched function in

running kernel

Modified Old Function to

insert a JMP instruction

Patch

applied

stop_machine() : disable CPU

interruption and set the

patching process in High

priority scheduling mode

Patched Function

Check kernel stack thread

Patch creation magic (2/2)

Ksplice - Alternatives

▪ Kpatch (RedHat) / kGraft (Suse):

▪ Uses ftrace to do the patching

▪ Kpatch uses stop_machine() to load new code (like Ksplice)

▪ Kgraft works on thread level, as soon as the threads reaches a point it switches to a new function

▪ KernelCare (CloudLinux)

▪ Proprietary software

Kpatch

▪ Kpatch > LivePatch (RedHat)

April 2015

Kernel 4.0

release –

Foundations for

LIVE PATCHING

May 2014

Submitted

Linux Kernel

Mainline

February 2014

Public release

GPLv2

June 2014

RHEL 7 – Preview

Canonical has a service called LivePatch that takes advantage of Kernel 4.x LIVE PATCHING

Ksplice – Web Management tool

▪ Webinterface available at: http://uptrack.ksplice.com or https://status-ksplice.oracle.com

▪

Ksplice – Web Management tool

▪ Webinterface available at: http://uptrack.ksplice.com or https://status-ksplice.oracle.com

Ksplice - Uptrack offline client

▪ Remove the need for servers to connect to “public” Ksplice server

▪ Create a local YUM mirror of Ksplice mirror and allow your servers to connect to it

▪ Or, Oracle bundles all available Ksplice patches for each supported kernel version to one RPM package and install in every server using a pen drive

▪ A new patch is available, a new RPM is available

▪ You cannot use the Web management tool.

▪ Details for offline client: https://docs.oracle.com/cd/E37670_01/E37355/html/ol_olcsetup_ksplice.html

Ksplice - Ksplice SNMP Plugin

▪ Use Oracle Enterprise Manager 12c/13c(?) to monitor status of Ksplice on your systems.

▪ Query the status of Ksplice using SNMP command snmpwalk:

[root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatusKSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate

▪ [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable

▪ KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd] KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k] KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]

Details: http://docs.oracle.com/cd/E37670_01/E39380/html/ch01s11s01.html

Watch and learn..to tune wait events

Watch and learn..to tune wait events