Squeezing Water from Stone: KornShell in 2019 · 2019. 10. 11. · What this talk is about ? It’s about AT&T KornShell (ksh, not mksh or pdksh). Brief status update from new upstream

Squeezing Water from Stone:

KornShell in 2019

Siteshwar Vashisht@SiteshwarV

main() { printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);}

● Winner of IOCCC (The International Obfuscated C Code Contest) one line entry in 1987.

● Termed as best one liner ever received.● Written by David Korn.

What this talk is about ?● It’s about AT&T KornShell (ksh, not mksh or pdksh).● Brief status update from new upstream maintainers (Kurtis Rader and

Siteshwar Vashisht).● No deep dive in the source code.● It is slightly extended version of my talk from FOSDEM 2019.

/whois situ● Siteshwar Vashisht.● Based in Brno, Czech Republic.● Maintainer of bash and ksh at Red Hat.● Previously involved with fish shell and Sailfish OS.● Current upstream maintainer of ksh.

What is KornShell ?● Direct descendant of Bourne shell.● Initial development based on original bourne shell.● Roots back to late 70’s and early 80’s.

Spot the bug!#!/bin/bash

echo $'hello world' | while read foo

do

bar="$foo"

done

echo "bar: " "$bar"

$bar is not available outside loop. Bash forks last command in a new process and $bar is lost.

Advantages● Superior language specification (Support for Discipline functions,

namespaces, compound data type, enums etc.).● Better POSIX compliance.● Fastest POSIX shell.

Show me the numbers!$ cat test.sh

for ((i=0;i

A 20 year old bug

State of ksh in 2017● Previous developers have left.● Codebase has origins going back to the 80s.● Old Tooling.● No revision control history.● Few meaningful comments in the source code.● Spaghetti code.● Bad test coverage.● Lots of old bugs.● And ...

How do you maintain the unmaintainable ?

Goals● Create a modern but backward compatible version of ksh.● Make it easier to contribute.● Improve tooling (build system, CI etc.).● Improve POSIX compliance.● Better test coverage.

Bring this codebase into the 21’st century.

One of my most productive days was throwing away 1,000 lines of code - Ken Thompson

We threw away more than 500,000 lines of code.

Lines of code (ksh93v-)

Lines of code (current)

What did we drop ?● Support for non-POSIX operating systems.● Reimplementation of POSIX functions.● Entire subsystems like the AST vmalloc and locale.● Lots of code that was not used by ksh.● All external commands (e.g., grep) other than ksh were dropped.

The project is now focused solely on ksh rather than providing replacements for commonly used commands found in BSD, Linux, and SysV.

Build System

● Nmake and iffe (If Feature Exists).● Build system was complicated.● Builds were slow.● We threw away old build system and switched to Meson. ● Build time reduced by a factor of 35.

real 5m15.922suser 4m12.355ssys 1m18.843s

Old Build System New Build System

real 0m9.239suser 0m11.290ssys 0m4.812s

Refactor Code

Refactor Code

It checks if either NV_LJUST or NV_RJUST attribute is set, but not both.

Here Be Dragons

Here Be Dragons

When ksh met gcc-9.0

Call for Developers

If you like to work on crazy problems, try refactoring ksh code.

https://github.com/att/ast

https://github.com/att/ast

“It was at first a small effort with about ten people, but it has grown gradually over time to 25 people. Most are researchers who create prototype Software.”

A quote from book “Practical Reusable Unix Software”

Computer Scientist vs Computer Programmer

A computer scientist is someone who looks on one side of the street while crossing one way street.A programmer always looks on both sides.

Better CI

Better CI

Test Coverage

Test Coverage

Test Coverage

All Red Hat internal tests are executed on every upstream build.

Call for Testers

You can be a crazy tester.

Coverity Defect Rate Graph

It’s simple to try

It’s simple to try

Call for Packagers

You don’t have to be a crazy packager.

$ meson build && ninja -C build

ksh-2020.0.0-beta1 was released in August 2019. If you don’t see it in your favorite distro, please rebase to newest version. Guidelines for packagers can be found at https://github.com/att/ast/wiki/Guidelines-for-Packagers

https://github.com/att/ast/wiki/Guidelines-for-Packagers

Package MaintainersBoyuan Yang - Debian

Cy Schubert - FreeBSD

Eli Schwartz - Archlinux

Mike Gilbert - Gentoo

Ryan Schmidt - MacPorts

Stig-Ørjan Smelror - Mageia

Is it too late to make a change ?

Change is Good. Better Late Than Never!

Thank you!