Video Terminal Evolution and The Future of Browsers

Evolution of The Clientand a viewable objects proposal

Thomas LynchThomas LynchCEO/founder Reasoning TechnologyCEO/founder Reasoning TechnologyPresented at NTUPresented at NTU2013-10-292013-10-29Thomas.Lynch@reasoningtechnology.comThomas.Lynch@reasoningtechnology.comwww.thomaswlynch.comwww.thomaswlynch.com

About Mr. Lynch● Participated in the design of a dozen Participated in the design of a dozen

microprocessors variously as individual designer, microprocessors variously as individual designer, tools developer, manager, and architect, [ including tools developer, manager, and architect, [ including the Am29000 (test), 29050 (12 instructions, fp the Am29000 (test), 29050 (12 instructions, fp support), Sonic (clock and smart card interface), support), Sonic (clock and smart card interface), Am187 (device library analysis), Am486, K5 (t-func Am187 (device library analysis), Am486, K5 (t-func develop, tech management), K6 (fp test platform), develop, tech management), K6 (fp test platform), K7 (fp and tech management), a PIC variation, K7 (fp and tech management), a PIC variation, Neptune, Chromatic FP, Quicksilver radio node Neptune, Chromatic FP, Quicksilver radio node (entire processor architecture, logic design, and (entire processor architecture, logic design, and tools) ]. tools) ].

About Mr. Lynch● I've had a consulting practice since 1996 where I I've had a consulting practice since 1996 where I

have worked with the startups Quicksilver have worked with the startups Quicksilver Technologies and NetEffect. Developed microcode Technologies and NetEffect. Developed microcode for Chromatic Computers. Created intellectual for Chromatic Computers. Created intellectual property for startups and for Intellectual Ventures. property for startups and for Intellectual Ventures. Created business plans including for Enreagle and Created business plans including for Enreagle and 21st Century Telephone. I proposed smart phone 21st Century Telephone. I proposed smart phone technology in the 90s, and social media technology technology in the 90s, and social media technology in the early 2000s.in the early 2000s.

About Mr. Lynch● 29 issued patents in the U.S. with a number of 29 issued patents in the U.S. with a number of

related international filings, and a similar related international filings, and a similar number of active applications. These area of: number of active applications. These area of: computer security; digital rights management; computer security; digital rights management; social media (relationship rings); high social media (relationship rings); high performance computing architecture, logic, performance computing architecture, logic, circuits, and packaging; user interface, and circuits, and packaging; user interface, and telecommunications.telecommunications.

Current Projects● Reasoning Technology – a new intuitive way to Reasoning Technology – a new intuitive way to

apply theorem proving to arguments in law.apply theorem proving to arguments in law.

● New Memory Architecture -New Memory Architecture -● Implications to databases (developing db)Implications to databases (developing db)● Implications to processor architectureImplications to processor architecture

● Computer SecurityComputer Security● A better way to limit viruses and other OS A better way to limit viruses and other OS

security threatssecurity threats

● New Media Compression AlgorithmNew Media Compression Algorithm

● New Discrete Optimization AlgorithmNew Discrete Optimization Algorithm

Evolution of the Terminal Device

First Video Game I Played● Strar Trek written by Mike Mayfield ● Each screen update printed out in

ASCII on the paper terminal.● Counted out distances manually

and used my SR-52 to calculate photo torpedo trajectories.

● Played it on the Boeing's Cybers (probably 7600) at work, and an HP3000 at school

The Teletype● Selonoid controlled cylinder jumped up and

slammed the letter on to the paper. Had a kick to it.

● Used an ink ribbon● Computer could type, or a person could type –

both about the same speed.● The computer typed out the game boards.

also ● Paper (that's right – paper) tape reader on the

side. The Star Trek game came from a friend on paper tape. We also had magnetic tape reader but mag tape was expensive and held more data than needed for such a program. Mag tape usage might also require operator assistance.

● The GE225 also used at the Army Corp of Engineers had this as the system console, and that machine was booted via paper tape.

DEC Writer● This had an ink ribbon and a print head that swept

across the paper. The print head had pins in it. It made a zzzz sound as it went across, and then the line was done. Much faster.

● After playing Star Trek for a few hours a box of paper would be used and be laying at your feet. If it wasn't cleared the printer carriage might start ripping the holes on the sides of the paper and then print over the same line repeatedly, perhaps at a bit of a diagonal.

● No hard push and kickback when you typed. The keyboard was soft, so typing was faster too.

● 132 characters wide, nicer than 80, though punch cards only had 1 + 71 + 8 columns (continuation mark, text, and comment) so often printouts only went part way across the page.

The Video Terminal● Plug-in replacement for the paper terminal (so you

also need printer to give results to the boss).● 80 by 25 line ASCII screen.● You will love green, later in amber also. ● Instant refresh in Star Trek - Don't have to wait for

the page to print to update the game board – yes!

Tektronix 4014● Internal electronics would accept an ASCII

sequence starting with an ESC character, a command byte, and two co-ordinates on the screen – and then direct the electron beam over the screen. There was no shadow mask, hence no pixels, just a solid line.

● High persistence phosphorous so the drawn lines would stay there until the user pushed the big erase button – which made a big flash and then the screen would be blank.

● Enabled drafting programs to run, where the draftsman could see in real time what the results looked like.

The Workstation● High Resolution Screen and graphics hardware to

drive it made raster graphics practical. It was intended to be used by an individual!

● Along with new CAD and word processing applications – applications that served an individual engineer rather than a project! [this and the prior point distinguish the workstation from the minicomputer]

● Possible to run programs locally● Server may now only be for maintenance and

central storage. Advent of Ethernet, RFS● X-windows – each window is now a terminal by

itself, simple mixing of text and graphics● Typically too expensive for an individual, also

predates visacalc.Apollo running Aegis

PC● Cheap Workstation that runs

visicalc – reached a wide audience● Visicalc did not require graphics –

which reduced the price considerably

● Simply Graphics● Proliferation of this hardware

facilitated the advent of a 'people's network'

Portable Computer – SubPC● Facilitated by the cell network and

Internet, WIFI/cell instead of Ethernet

● More powerful than the servers of yore.

● New Applications for individuals● Browsers is the Thin Client● Work Station like behavior for local

applications● Software maintenance needs and individuals

owning multiple devices has caused a renewed pressure to bring back the appearance of a server - leading to the cloud.

● Because multiple device ownership flips the person to computer ratio - I call it the era of subPC.

Network Situation – Full Circle?

Typical Configuration Star Trek Game Was Played On● RS232 point to point serial

communication, progenitor of USB. 9600 baud was typical.

● Server in the middle, each terminal plugs into a dedicated port.

● Peripheral interface cards have RS232 ports and plug into the peripheral bus of the computer.

● Control chip in the terminals are the progenitors of the graphics chip.

Computation must dominate over I/O, or progress will be very slow.

Typical Enterprise Workstation Network

● Individually Operated Programs Run Locally (such as the CAD tool)

● Servers come in farms instead of being a single machine.

● “Big Jobs” with little interactive input run on a server. More than one choice of server to chose from 'load balancing'.

● Software Installed Over the Network● Backups and Data Shared Over the

Network● A 'gateway' (not shown) provides

access to the Internet for email and browsing.

The biggest changes: a layer of hierarchy, the type of data on the network, i.e. the purpose of the network.

What Changes With HTML5?● Some suggest we go back to running all applications on a server, and that

the graphics and display work is done locally – just like in the old days.– This favors portable devices that don't have enough compute or battery power to

run individual purpose applications.● But ... just because we used a browser doesn't mean that the application

can't run locally. – Better than using a window – it solves the problem of differing GUI approaches

needed on different platforms today.– And about those mobile devices – trends in computing suggest that they will be

powerful enough that a trend back to remote running individual purpose applications will be at most temporary.

Evolution in Graphics Workflow

Star Trek ASCII Graphics● This is first few lines of Star Trek

in Basic. It has been extracted from an ancient paper tape.

● This is probably for the HP3000.● Apparently the world model is a

collection of arrays.

Trek Graphics● Array of characters

keeps their information● Map of world has

layers, each layer has a character describing something about that location

1. Scan world(i+wi, j+wi, z) into display(i+di0, j+dj0). Look across z layers for combination events.

2.Update information into information areas in display.3.Scan display array one row at a time, sending string out to RS232 port. At end of row

send CR-LF, go to next row.4. If enterprise is dead, goto 5,else wait for user input, parse it and update the world and

information.5.Terminate the job (process)

Textronics 4014 Graphics● If you stare at a green screen all day and then look at

a white wall, the wall looks pink.● So called vector graphics or wire frame graphics. The

terminal excepted draw line segment commands, so you only saw the outline.

● Initially you could not erase a line, it physically persisted on the phosphorus – the whole screen was erased at once.

Tektronics 4014 Graphics● Display list created from

your model, or it may be the model

● Homogenous co-ordinate systems so matrix transforms can do displacement

● View step transforms display list from view point, and scales to view co-ordinates

● Clipping chops off things off screen

● Finally break display list down to escape commands

● Libraries to help with this:● GCS● Plot10

Modern Graphics● Display list is now triangles instead of vectors● View now includes lighting and shading, possibly even ray tracing● Display buffer is now binary and there is no step of breaking it down to ASCII

commands● Hardware acceleration – initially this was very specific to the graphics problem, but the

latest generation are SIMD machines (like the old Cray machines)

But Wait a Minute ..What of Layout?But Wait a Minute ..What of Layout?

What of Layout● Notice in Star Trek, we had a concept of layout. We had an area on the screen for

the ASCII graphics, and other areas for information.

● We lost this when we went to the display list. Our world model had no place for it.

● The rest of the world did not forget, we have a parallel evolution in the “GUI”: ● Windows (Xerox PARC)● X-windows widgets● Html● Css

X-windows – The Happenings● A graphics 'server'

● Events – a key press throws things at programs

● Call backs – wake up threads when something happens

● Multithreading – cooperating routines all running at the same time

● Widgets – things that display, have data, and can throw events or make callbacks

Html Browser– It is a Client● Conceived around the concept of hyper text. Bunches of little text pieces, with

underlined areas that could be clicked on to be expanded.

● Added basic text formatting and started to look something like word processor output.

● Because a language for storefronts on the Internet, so it integrated to a database and 'did things' in response to user input other than just show more text.

● Added media players

● Dynamic html, and now html5 – The browser becomes a sort of X-window, but better in some ways:● Universally portable and standard compliance across vendors.● Does its own screen formatting in ways familiar to everyone● Runs programs, e.g. javascript so it has become a client in itself.

● Note that our application is back to sending characters out the port (default 8080)

HTML/CSS/script vs. Widgets

Both CSS and Widgets Assist With Layout, but they are different:

● Widget containers own other widgets. The contained widget then lays out inside the area for the container.

● HTML has <div> blocks, where the items in them layout within the <div> is in a sense analogous to a widget container.

● Widget contains code that does things, takes actions, throws events.

● HTML supports java script, which may manipulate the graphics and post events

Though notice, Javascript is not a good language for writing an application in. Hence we have frame works becoming available, e.g. Ruby on Rails – templates and filling them in. And Wt – html rendering widgets for C++.

Problems I had with Wt● Some mechanical stuff not related to the concept of using Widgets. -over use of static

methods, singleton classes, no iterators over containers, some other things.

● Widgets keep copies of the data – so there is a lot of data duplication.

● Widgets manage their own memory, so the data must live on the heap – and the program can't manage it the way it would like.

● Widgets exist in a tree hierarchy, but they do not explicitly recognize this. Hence, we throw out efficient data structures in favor of implicit trees.

● Widgets come in a library. One becomes a library developer to develop fundamentally new ones – and that requires fitting into a complex framework.

A ProposalA Proposal

Display Objects

● similar in flavor to iterators for containers but its job is to display the 'data object'.

● like an iterator the display object knows about its data object, and it has a standard interface -though in this case for causing the data object to be displayed rather than iterated over.

● as for an iterator the implementation is specific to the data class. Hence the display object knows where to find the data fields and how to use the data object.

● The display object may throw various events depending on its implementation.

● When used with a browser the display object generates java script and receives posts in the form of commands from a layout object.

● Library of existing display objects to inherit from (analogous to the widget library)

Layout Objects

● Communicates with display objects.

● Gather preferred size, and size limits, dictate actual size.

● Provides environment information and CSS styles

● For html browser application gathers java script output.

● Notifies the display object of posts.

● Layout Objects may be nested, i.e. the Layout object is a type of display object, just can't be a root node.

Open Source Project● If anyone is interested in helping develop this

let me know.