User Interface Software Technology Easier to use, but harder to program –Multiprogramming –Real-time programming –Robustness –Testability –Modularization.

User Interface Software Technology Easier to use, but harder to program Multiprogramming Real-time programming Robustness Testability Modularization Need to support iterative design and testing Software tools User Interface Software Concepts Dialogue independence Interface (syntax) vs. application (semantics, business logic) Model-View-Controller architecture Design levels: conceptual, semantic, syntactic, lexical Generate UI from abstract description Early research in user interface management system (UIMS) and user interface description languages (UIDL) User interface software tools Goal: Support dialogue independence and separation of programming roles User Interface Software Tools Toolkit Application Window system/graphics package Operating system Hardware UIMS Window Systems Manage shared resources (pixels, keyboard, mouse), provide clipping, translation Tells client when exposed, resized, or other damage Could implement in different places Single built-in window manager or allow separate Dispatcher (N.B. Security classifications shown are simulated only.) AccessWindowMouseEvent (Event *e) {... FileWindowMouseEvent (Event *e) { If (e->type==MS_LEFT)... Toolkits Library of reusable routines for implementing human-computer interface features Basic components Set of widgets "Intrinsics" Does not try to describe or support user interface as a whole Standard look and feel Encourage consistency But imposes limits on interaction style Toolkits (cont'd) Examples Visual Studio (Windows) Interface Builder (Macintosh) GTK (X/Unix/Linux) Based on controls or "widgets" (N.B. Security classifications shown are simulated only.) AccessCheckBox (Boolean val) ClassifTypein (char *val) DisplayMsgFile () UIMS Higher level than procedure library May include main program, provide external control of application Call semantics/business logic as subroutines Message to component software Design time, run time, and evaluation time tools Module partitioning: Runtime vs. interface design time partitioning User Interface Description Languages How does the user interface designer describe the desired user interface to the UIMS or toolkit? Layout Behavior With a User Interface Description Language (UIDL) May be interactive Generate UI from abstract description UIDL describes user-visible behavior, but not implementation Other uses of UIDL (evaluation) User Interface Description Language Designer creates it, UIMS executes it Specific language or technique suitable for describing user interfaces Describe user-visible behavior, but not implementation Goal: Generate a UI from high-level, abstract description Platform-independent if possible (Windows, Mac, X) Platform-type-independent if possible (desktop, cellphone, voice)! Model-based UI development Current GUI practice Has evolved from general purpose UIDL into small, stable set of widgets and classes Change with new UI styles, portability needs Choice of design language at heart of UIMS Based on techniques for specifying static languages Both can be modified for interactive languages BNF send-cmd::= SEND DRAFTNAME RECIPNAME State Transition Diagram Alternate Form of State Transition Diagram start: SEND ->getdr getdr: DRAFTNAME ->getre getre: RECIPNAME ->end Basic Classes of UIDLs startgetdrgetreend SENDDRAFTNAMERECIPNAME Early UIMS/UIDL Research Language-based Menu Network [Kasik] BNF, grammar-based [Olsen] State transition diagram [Newman] Event language Declarative Non-language-based Interactive builder (bottom up) Current state of practice, use GUI techniques to specify GUI Plug together components with pre-defined interactive behavior Automatic generation (top down) [Foley] By demonstration [Myers] Other categories Constraint-based [Hudson] User-oriented [Moran] Example: Visual Programming for UI Layout vs. Behavior UIMS, with state diagrams for behavior ] Visual Basic, pre-programmed behavior Example: State Diagram as UIDL Command-driven Graphical Report Editor UI Evaluation Directly from UIDL Evaluate interface without building system Emerging area, but lack underlying formal knowledge of UI design Analyze user interface to detect specific properties [Darlington et al.] Apply human performance models [Card] Automated screen design critique [Tullis] Prove safety-critical properties Screen mockup, prototyping tools Slide show" of canned screens based on user inputs UIMS sans application (use stubs for semantics) Visual Basic, Powerpoint, plain HTML also used this way UI Tools: Standardization Early: Many desktop platforms No standardization Current: ~3 major platforms, standard widgets Settled into stable toolkits, widget libraries, GUI class libraries UIMS seems like overkill Emerging: Mobile, automobile Diverse platforms, no standardization Wider range of interaction techniques Model-based development, UIMS redux Future: Same issue for new non-WIMP interaction styles Still wider range of possible interactions UI Tools: Trends UI toolkits, libraries Become important, integral part of software development tool sets e.g., MacOS UI features are well integrated Good uptake despite unusual programming language Web browser as UI platform Cloud computing Chrome-only OS New tools for web UIs, but still clumsy, poorly integrated Making cross platform toolkits less significant UI Tools: Mobile Devices Basic HCI development and design principles Largely GUI, not new interaction style + gestures, GPS,... Use context: Full attention or not? Development tools Early: Clumsy tools, restricted UI designs Much like early GUIs, function key-driven (pre-mouse) GUI Current: UIs and tools very similar to standard GUI IOS: Interface Builder (Mac) Android: Java + Swing (minor differences) UI Consistency Low level consistency High level models, UIDL could help Example: IMAP mail clients Future Directions: Emerging Interaction Styles Parallel, Highly-interactive (non-WIMP) Lightweight, Context-aware, Non-command Continuous (+ Discrete) Tangible Interfaces Microprocessors, 3D forms Ubicomp, Pervasive, Context-aware Sensors, networks Reality-based Interaction Brain-computer Interaction Implications for Software CurrentFuture Single thread I/OParallel, asynchronous dialogues; may be interrelated Discrete tokensContinuous inputs and responses (plus discrete) Precise tokensProbabilistic input, not easily tokenized Sequence, not timeReal-time requirements, deadline-based Explicit user commands Passive monitoring of user PMIW: A UIMS for VR, Non-WIMP Handle continuous explicitly in language Could handle with events as usual, but wrong model for non-WIMP Want continuous as first-class element of language Software for Emerging Interaction Style Language to describe and program fine-grained aspects of non-WIMP interaction Basis: Essence of non-WIMP = set of continuous relationships, in parallel, most are temporary Combine data-flow component for continuous + event-based for discrete Discrete can enable/disable the continuous links Separate non-WIMP interaction into 2 components Each can exploit existing approaches Provide framework to connect the 2 Keep model simple enough for fast run-time Support VR interfaces directly User Interface Software User Interface Management System (UIMS) Designer specifies user interface, UIMS implements it Specification technique is key May be interactive State transition diagram-based UIMS Prefer state diagrams to BNF because they make time sequence of dialogue explicit Develop state diagram-based technique that identifies and separately specifies the three levels of the interface Semantic Level Syntactic Level Lexical Level Background Two components + communication framework Discrete Existing UIMS, UIDL technology from WIMP BNF, grammar-based, state transition diagrams, event response,.... Continuous Like data-flow graph or set of one-way constraints between inputs and outputs Constraints: 1-way, 2-way, continuous 3-D geometry Plus very high performance Plus time management for video-driven Plus ability to re-wire graph from user inputs Why Not Just WYSIWYG? Why not XCode Interface Builder or Visual Basic? Visual layout of objects in UI But not new interactive behaviors 2 classes of VL's VL for static visual objects VL for abstract, non-visual (e.g., sequence, behavior) Most current UI toolkits provide widgets Can change position, size, appearance But canned behavior Need new language for behavior Toward a Language Basic structure of non-WIMP interaction Claim: A set of continuous relationships, most of which are temporary Handle continuous explicitly in language Current models typically based on tokens or events Quantize continuous into change-value or motion events and handle as discrete But events wrong model for parts of non-WIMP Data-flow graph Continuous variables Continuous functions Like a plugboard, wiring diagram, 1-way constraints Implicitly parallel Model Two-part description of user interaction Graph of functional relationships among continuous variables (few typically active at a time) Discrete event handlers (which can turn the continuous relationships on and off) Communication paths Discrete activate/deactivate links by set/clear Conditions Link recognize pattern, trigger discrete event Both set/test arbitrary shared UI variables Language Set of continuous Variables Some connected to input devices, outputs, application semantics, and some for communication within UI Set of Links Function from continuous variable to continuous variable Conditions Link attached to Condition, allows turn on/off Set of EventHandlers Respond to discrete input events, produce outputs, set syntactic-level variables, call application procedures, set/clear Condition Object-oriented framework Link, Variable, EventHandler in separate class hierarchy Basic UIMS in base classes: Variable, Link, EventHandler Example Conventional WIMP slider to show notation We view as continuous relationships, but activated by mouse down/up events Alternate (Integrated) Form Conceptually each state has entire data-flow graph When system enters state, begins executing that data-flow graph and continues until exit state Diagram = transitions between whole data-flow graphs Apt for moded continuous operations Zoomed In Difficult to fit integrated form in single picture Interactive zooming editor Even better: Rapid continuous zooming (PAD++) or head-coupled zooming (pre-screen projection) Previous diagram, zoomed in on the first state Enclosed data-flow diagram now visible and editable Grab in 3-D Grab and drag object with hand in 3-D Common, simple interaction in VR Diamond cursor permanently attached to user's hand Ugly object can be grabbed and moved UIDL for Grab in 3-D Grab object by holding Button 1 While button held, object position = cursor position When button released, relationship ceases (but cursor still follows user's hand) Hinged Arm User can grab arm and move in 3D Left end always fixed to base column, as if hinged Arm pivots to follow hand cursor UIDL for Hinged Arm State change when user grabs arm (activates linkc1), and releases arm (deactivates linkc1) Hand (polhemus1) always drives cursor position Linkc1 connects cursor position to arm rotation continuously But active only while user grasping arm Two-Jointed Arm User can grab and move the first (proximal) segment of the arm as in previous example Second (distal) segment hinged at tip of proximal User can grab distal and rotate wrt tip of proximal UIDL for Two-Jointed Arm Linkc1 active when hand cursor controlling rotation of proximal segment (GRASPED1 condition) Linkc2 active when hand controlling distal (GRASPED2) Language clearly shows: Depending on state, hand position sometimes controls rot1 and sometimes rot2 World with Many Arms Two instances of two-jointed + 24 of one-jointed One-jointed arms point to proximal/distal tips of two-jointed, can turn on/off in groups Use to demonstrate performance PMIW User Interface Management System Software model and language for inventing new non-WIMP interaction techniques Based on: Essence of a non-WIMP dialogue is a set of mostly temporary continuous relationships Attempt to capture formal structure of non-WIMP as previous techniques captured command-line, menu-based, moded, event-based Higher level user interface description language constructs for non- WIMP Especially VR where severe performance requirements Demonstrate new language won't cost VR performance Penalties paid at compile-time Graphical editor, run-time UIMS Tangible User Interfaces Simple, transparent mechanical structures Use knowledge of physical world to operate Augment physical objects with digital meaning Combine physical + digital representations Exploit advantages of each Technology for TUI Sensors Produce signal in response to change in surroundings Actuators Produce physical change in response to signal Microcontroller Communicates with devices and with main computer Other technologies RFID Computer vision ... Wider range of I/O events than GUI Development methodology, tools None! A Visual Language for Modeling Tangible User Interfaces [Shaer] TAC paradigm Each TUI consists of token within a constraint Same object may sometimes be token, sometimes constraint Two tier model fits well Dialogue (states, storyboard) Interaction (especially continuous) TUIML Eric Abouaf Model-View-Controller Design pattern, originally Smalltalk Found, in some form, in most UI toolkits Matches dialogue independence, syntactic/semantic split USER ModelControllerView