讓Linux 核心更安全 檢測並修補安全漏洞 · 2020. 8. 21. · Tech & Telecom 35,000+ Developers Contributing Code 170+ Open Source Projects $16B Shared Value ... Open Source

讓 Linux 核心更安全 – 檢測並修補安全漏洞透過軟體開發生命週期管理核心安全漏洞

SZ Lin (林上智)

12th August, 2020

CYBERSEC 2020

Software R&D Engineer, Software Development Dept.

About Me - 林上智 (SZ LIN)

178F 8338 B314 01E3 04FC

44BA A959 B38A 9561 F3F9

Software Engineer, (In-house) Consultant

Embedded Linux Design and Development

- IIoT platform developer

- Civil Infrastructure Platform – Linux Foundation Project• Former Kernel Team Chair

• Technical Steering Committee Member

Open Source Development and Governance

- Debian Developer (pkg-security-team)

- OpenChain Project Governing Board Member

Cybersecurity

- CISSP – ISSAP, CSSLP

- ISA/ IEC 62443 Cybersecurity Expert

- Security Workgroup member in CIP project

> 80 % > 75 % 100 %> 95 %

img src: https://kernel.org

src: https://www.linuxfoundation.org/about/

of the top one

million domains

run with Linux

of cloud-enabled

enterprises report

using Linux as

their primary cloud

platform

of new

smartphones sold

run Android, which

is based on the

Linux kernel

of the top 500

supercomputers in

the world run on

Linux

Before Using Linux KernelSomething you should know

5

CopyrightCopyright PatentPatent

A patent gives its owner the right to

exclude others from making, using,

selling, and importing an invention

for a limited period of time, usually

twenty years.

src: https://en.wikipedia.org/wiki/Patent

Copyright is a legal right, that grants

the creator of an original work

exclusive rights to determine whether,

and under what conditions, this

original work may be used by others

src: https://en.wikipedia.org/wiki/Copyright

6

CopyrightCopyright PatentPatent

A patent gives its owner the right to

exclude others from making, using,

selling, and importing an invention

for a limited period of time, usually

twenty years.

src: https://en.wikipedia.org/wiki/Patent

Copyright is a legal right, that grants

the creator of an original work

exclusive rights to determine whether,

and under what conditions, this

original work may be used by others

src: https://en.wikipedia.org/wiki/Copyright

Context

7

1400+Members From

41 Countries

80%of Fortune 100

Tech & Telecom

35,000+Developers

Contributing Code

170+Open Source

Projects

$16BShared

Value

Linux Foundation

8

The OpenChain Project defines the key requirements

of a quality open source compliance program [1].

src:https://www.iso.org/standard/81039.html

12

User ApplicationsUser Applications

GNU/ Linux

GNU C libraryGNU C library

Init system

UserSpace

KernelSpace

Hardware and peripheral devices

Architecture-dependent firmwareArchitecture-dependent firmware

BootloaderBootloader

KernelKernelSystem call interfaceSystem call interface

ToolchainToolchain

Root filesystem

More info: Using open source software

to build an industrial-grade embedded

Linux platform from scratch

Open Source Summit Japan, 2019 [57]

End of LTS

13

Linux Kernel Releases

Mainline

Stable

(linux-stable-4.4)

v4.4

Stable

(linux-stable-4.19)

6+? years

v4.5 v4.19 v5.x

EOL

v4.4.x v4.4.y v4.4.z

v4.19.a v4.19.bimg src: https://en.wikipedia.org/wiki/Linux_kernel_version_history

End of LTS

6+? years

27.8 60-90 Day 66,492 3,386,34721,074

Mainline Kernel

Release CycleMillion Lines Files Lines of New Codes

in 2019Different Authors

14

src: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Git-Stats-EOY2019

img src: https://kernel.org

Supply Chain Risk Management

Practices for Federal Information

Systems and Organizations

Special Publication 800-161 [4]

SM-9: Security requirements for

externally provided components

ISA/ IEC 62443-4-1 [5] NERCCIP-010-2 [6]

Configuration Change Management

and Vulnerability Assessments

img src: https://pixabay.com/illustrations/policies-standards-compliance-4720824/

15

src: https://www.ithome.com.tw/news/138633

2020-07-07發表

16

How to Manage

Vulnerabilities in

Linux Kernel?

17

Costs to Fix Software Defects at Different

Stages of SDLC [7]

1x5x

10x

15x

30x

0

5

10

15

20

25

30

35

RequirementsGathering and

Analysis/ ArchitecturalDesign

Coding/Unit Test Integration andComponent/RAISE

System Test

Early CustomerFeedback/Beta Test

Programs

Post-productRelease

18

X is a normalized unit of cost and can be expressed terms of person-hours, dollars, etc.

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

19

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

20

Scope

ScheduleResources

Good enough

principleKISS principle

Core technology

identification

Requirements Analysis

It’s imperative to collect, analyze, identify requirements for Linux

kernel and its configuration, it also reduces the unnecessary

maintenance effort related to security. Moreover, it provides

information for us to choose proper kernel source to fulfill our

requirements.

21

Requirements for the Civil Infrastructure Systems [8]

Industrial Grade

• Reliability

• Functional Safety

• Security

• Real-time capabilities

Sustainability

Security

• Security & vunerability managment

• Firmware updates

• Minimize risk of regressions

This has to be achieve with …

Development time

Shorter development times for more complex

systems

Maintenance costs

Low maintenance costs for commonly uses

software components

Low commissioning and update costs

Development costs

Don‘t re-invent the wheel

• Product life-cycles

of 10 – 60 years

22

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

23

Choose Proper Linux Kernel

only from trusted sites

24

CategoryLatest

versionTarget Application Maintainer

Linux kernel 5.8• Performance

• Resource Limited [9] [10]Kernel.org

Preempt RT

kernel5.6

• Real-time

• Functional safety

• Resource Limited

Real Time Linux

collaborative project

*Real-time application [11][12]

25

*Grsecurity [13]

SoC Board Support Package Kernel

• Kernel version depends on SoC vendors

– Well made but not well maintained

• Contain lots of in-house patches

– Errata patches

– Specific feature patches

– …

• Different SoC might use different versions of kernel

• The lifetime is unsure

26

LTS: Long Term Stable Kernel [3]

Extend software uptime for stable kernel

• Only accept bug fixes and security fixes

img: https://www.kernel.org/category/releases.html

Retrieved 7th August

27

LTSI: Long Term Support Initiative [14]

• Linux Foundation collaborative project – Based on LTS

– Add another chance to include further patches on top of LTS

– Auto Test framework

– Same lifetime with LTS (yearly release and 2 years life time)

28

CIP (Civil Infrastructure Platform) [16]

• Linux Foundation collaborative project – Support kernel and core package

– Auto Test framework

– Maintenance period• 10 years and more (10-20 years)

29

More info: CIP Kernel Team Activities to Accomplish Super Long Term Support

Embedded Linux Conference, 2020 [17]

CIP SLTS Kernel Releases

Mainline

Stable (linux-stable-4.4)

4.4

CIP SLTS 4.4 (linux-4.4.y-cip)

CIP SLTS 4.19 (linux-4.19.y-cip)

End of LTS

Stable (linux-stable-4.19)

Maintained by CIP

Maintained by

CIP Kernel

Maintainers

4.19

10 years

6 years

4 years

End of CIP SLTS

5.x

10 years

6 years

4 years

30

Speed and Efficiency : focus on differentiating parts

31

Linux Kernel Source Comparison Table

Version

Maintenance

Period

(years)

FeaturesLatest

Version

Supported

Real-time

kernel

Maintainer

SoC

BSP kernel? Bug fixes ? N SoC vendor kernel team

LTS

kernel2 ~ ?

• Bug fixes

• Security fixes 5.4 N Kernel.org

LTSI kernel 2 ~ ?

• Bug fixes

• Security fixes

• Specific features

• New features

4.14 NLTSI

(Linux Foundation Projects)

CIP

kernel10 +

• Bug fixes

• Security fixes

• Specific features

• New features

4.19 YCIP

(Linux Foundation Projects)

32

ELISA: Safety-Critical Systems [17]

• Linux Foundation collaborative project

– Build and certify Linux-based safety-critical applications

– Define and maintain a common set of tools and processes

• SIL2LinuxMP [18] project and the Linux Foundation’s Real-Time Linux project

– IEC 61508

33

Year 2038 Problem [19][20]

• The time_t datatype is a data type in the ISO C library and kernel structure defined for storing system time values.

• 32-bit system can represent dates from Dec 13 1901

Jan 19th 2038

• It causes integer overflowing on – 03:14:08 UTC 19 January 2038

34

Don’t choose rolling version

unless necessary

v4.4.1

Security fixesSecurity fixes

Bug fixesBug fixes

Upstream

rolling version

35

v4.4.2 v4.4.3

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

36

Upstream First

37

Kernel inside the organization

Upstream

38

v4.4.1

Kernel inside the organizationIn-house security or

bug patches

In-house security or

bug patches

Upstream

39

v4.4.1

v4.4.1

Security fixesSecurity fixes

Kernel inside the organization

Bug fixesBug fixes

Upstream

40

v4.4.1 v4.4.2

v4.4.1

In-house security or

bug patches

In-house security or

bug patches

Security fixesSecurity fixes

Kernel inside the organizationIn-house security or

bug patches

In-house security or

bug patches

Bug fixesBug fixes

Upstream

41

v4.4.1 v4.4.2

v4.4.1 v4.4.2

Security fixesSecurity fixes

Kernel inside the organizationIn-house security or

bug patches

In-house security or

bug patches

Bug fixesBug fixes

Upstream

42

v4.4.1 v4.4.2

Security fixesSecurity fixes

v4.4.3

v4.4.1 v4.4.2 v4.4.3

Security fixesSecurity fixes

Kernel inside the organizationIn-house security or

bug patches

In-house security or

bug patches

Bug fixesBug fixes

Upstream

43

v4.4.1 v4.4.2

Security fixesSecurity fixes

v4.4.3

v4.4.1 v4.4.2 v4.4.3

Security fixesSecurity fixes

Kernel inside the organizationIn-house security or

bug patches

In-house security or

bug patches

Bug fixesBug fixes

Upstream

44

v4.4.1 v4.4.2

Security fixesSecurity fixes

v4.4.3

v4.4.1 v4.4.2 v4.4.3

• The project shares its results with the upstream

• The project fulfills longer time maintenance and

security fixes

• The project develops their code very quickly

• The project faces difficulties to backport upstream

patches due to conflicts as time goes by

45

Kernel Hardening –

Configuration OptimizationSecure the system by reducing its attack surface

46

47

48

49

50

51

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

52

For Stable Kernel Maintenance

• Automated Linux Kernel Testing [22][23]

– Detect, bisect, report and fix regressions on upstream Kernel trees before release

– Short tests on many configurations

53

img src: https://kernelci.org/

img src: https://kernelci.org/

54

55

Reproducible Builds [25]

• Create an independently-verifiable path from source to binary

– Ensure builds have identical results

– Act as part of a chain of trust

– Prove the source code has not been tampered/modified

56

Continuous Integration • Jenkins [26]

• Jenkins X [27]

Continuous Delivery/ Deployment • LAVA 2 [28]

Distributed compiler service • icecc [29]

• GOMA [30][31]

• distcc [32]

Test Case Management • Jenkins

• LAVA 2

Version Control • Git with gitlab [33]

Static Program Analysis • checkpatch.pl [34]

• sparse [35][36]

• smatch [37]

Dynamic Program Analysis • Profiling tools [38]

Vulnerability Scanning • OpenVAS [39]

• Vuls [40]

Fuzzing Testing • Syzkaller [41]

• Trinity [42]

• perf_fuzzer [43]

More info:

Building, Deploying and Testing an

Industrial Linux Platform

Open Source Summit Japan 2017 [44]

57

SDLCSoftware

Development Life Cycle

Requirement Analysis

Design

ImplementationTesting

Maintenance / Evolution

58

0

400

800

1200v5.4

v4.19

v4.14v4.9

v4.4

Commit Counts per Month

Note: If a patch has an original patch, the date of the patch is that of the original one.

59

v4.19

v4.4

60

v4.9

v4.14

• cve-search [45]

• nvdtools [46]

• Distribution CVE tracker

• National vulnerability database [47]

• Upstream issue tracker or forum

Vulnerability Scanning – Component Level

61

Vulnerability Scanning – System Level

Security

Quick response in

resolving CVE/

vulnerabilities and

attacks in platform

Daily test for CVE

…

Daily test for CVE

…

62

Vulnerability Management Framework

Dependency-Track [49]

SW360 [48]

63

Vulnerability Scanning – Source Code Level

64

• This project tracks the status of security issues, identified by CVE

ID, in mainline, stable, and other configured branches.

Introduction to "cip-kernel-sec”

65

Issue Format - YAML

66

Show via Web I/F

Mainline/LTS

cip-kernel-sec

Webview Command line view

Gather CVE Information for Kernel

Show via Command Line

67

cip-kernel-sec Web View

6868

Linux Kernel Vulnerabilities = Bugs != CVEs

69

src: https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/

70

71

src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076

72

src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076

Community

Collaboration

Different approach for

multiple target applicationsPreparedness Planning

Testing and

well-maintenance

Conclusion

© Moxa Inc. All rights reserved.

Thank You

[1] https://www.openchainproject.org/

[2] https://www.iso.org/standard/81039.html

[3] https://www.kernel.org/

[4] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161.pdf

[5] https://webstore.iec.ch/preview/info_iec62443-4-1%7Bed1.0%7Den.pdf

[6] https://www.nerc.com/pa/Stand/Reliability%20Standards/CIP-010-2.pdf

[7] https://www.nist.gov/system/files/documents/director/planning/report02-

3.pdf

[8] Industrial-grade Open Source Base Layer Development, Yoshitake

Kobayashi, Urs Gleim.

Referneces

[9] https://tiny.wiki.kernel.org/start

[10] https://bootlin.com/pub/conferences/2017/jdll/opdenacker-embedded-

linux-in-less-than-4mb-of-ram/opdenacker-embedded-linux-in-less-than-

4mb-of-ram.pdf

[11] https://xenomai.org/

[12] https://www.rtai.org/

[13] https://grsecurity.net/

[14] https://ltsi.linuxfoundation.org/

[15] https://events.linuxfoundation.org/wp-content/uploads/2017/11/Using-

Linux-for-Long-Term-Community-Status-and-the-Way-We-Go-OSS-

Tsugikazu-Shibata.pdf

Referneces

[16] https://www.cip-project.org/

[17] https://static.sched.com/hosted_files/ossna2020/d0/OSSNA2020-CIPKernelTeam-2.pdf

[17] https://elisa.tech/

[18] http://www.osadl.org/SIL2LinuxMP.sil2-linux-project.0.html

[19] http://elinux.org/images/6/6e/End_of_Time_--_Embedded_Linux_Conference_2015.pdf

[20] https://en.wikipedia.org/wiki/Year_2038_problem

[21] www.cvedetails.com/vulnerability-list.php?vendor_id=33&product_id=47&version_id=261041&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=2019&month=0&cweid=0&order=3&trc=72&sha=53735ab937bcf3686d34f3999d8e47f304466007

Referneces

[22] https://kernelci.org/

[23] https://fosdem.org/2019/schedule/event/kernelci_a_new_dawn/attachments/slides/3300/export/events/attachments/kernelci_a_new_dawn/slides/3300/gtucker_kernelci_fosdem_2019_v2_3_1024x768.pdf

[24] https://kernelci.org/build/stable/branch/linux-4.19.y/kernel/v4.19.138/

[25] https://reproducible-builds.org/

[26] https://jenkins.io

[27] https://jenkins.io/projects/jenkins-x/

[28] https://validation.linaro.org/static/docs/v2/#

[29] https://github.com/icecc

[30] https://chromium.googlesource.com/infra/goma/server/

[31] https://chromium.googlesource.com/infra/goma/client

[32] https://github.com/distcc/distcc

[33] https://about.gitlab.com/

Referneces

[34] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/checkpatch.pl

[35] http://sparse.wiki.kernel.org/

[36] https://git.kernel.org/pub/scm/devel/sparse/sparse.git

[37] http://smatch.sourceforge.net/

[38] https://perf.wiki.kernel.org/index.php/Main_Page

[39] http://www.openvas.org/

[40] https://vuls.io/

[41] https://github.com/google/syzkaller

[42] http://codemonkey.org.uk/projects/trinity/

[43] http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/

Referneces

[44] http://events.linuxfoundation.org/sites/events/files/slides/Build

ing%2C%20Deploying%20and%20Testing%20an%20Industrial%20Linux%

20Platform.pdf

[45] https://github.com/cve-search/cve-search

[46] https://github.com/facebookincubator/nvdtools

[47] https://nvd.nist.gov/

[48] https://www.eclipse.org/sw360/

[49] https://dependencytrack.org/

[50] https://www.cvedetails.com/version/261041/Linux-Linux-Kernel-4.19.html

[51] https://www.cvedetails.com/version/230587/Linux-Linux-Kernel-4.14.html

[52] https://www.cvedetails.com/version/205966/Linux-Linux-Kernel-4.9.html

[53] https://www.cvedetails.com/version/190796/Linux-Linux-Kernel-4.4.html

[54] https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec

Referneces

[55] https://icss20.sched.com/event/ZjMw/managing-vulnerabilities-in-open-

source-components-in-ics

[56]

https://lore.kernel.org/lkml/[email protected]

.com/

[57] https://ossalsjp19.sched.com/event/OVsf/using-open-source-software-

to-build-an-industrial-grade-embedded-linux-platform-from-scratch-sz-lin-

moxa

Referneces