Michael Bernstein Principal Software Design Engineer Microsoft Corporation.

The New COM API for Accessibility and Automation in Windows 7

Michael BernsteinPrincipal Software Design EngineerMicrosoft Corporation

Programmatic UI Access: A Key Element of Accessibility

Enables developers to write code that:

Navigates between UI elements

Gathers information about the UI

Interacts with UI elements

Receives notifications when the UI changes

Accessibility

Programmatic Access

2

UI Automation (UIA):Programmatic Access in Windows

Successor to Microsoft Active Accessibility (MSAA) Creates a universal object model for accessing UI Designed based on experience with MSAA usage Supported on Windows Vista, XP and Server 2003 UIA COM Client API introduced in Windows 7

3

UI Automation Core MSAA

UIA Managed Client API UIA Provider API UIA COM Client

API

What’s new in Windows 7?

UIA Clients can now use COM instead of managed code

New Accessibility properties are available Native clients can register custom ‘proxies’

to make existing UI more accessible Developers can define custom Accessibility

properties and events beyond the official UIA list

Inspecting UIA Properties

demo

Context: Clients and Providers

Client-side API (C# or C++)

Client Provider

UIA Infrastructure

UIA

Provider-side interface definitions (C# or C++)

Test Automation and Assistive Technology Manufacturers

Application and Control Developers

TAKING ADVANTAGE OF THE NEW COM CLIENT API

Client ProviderUIA

SAMPLE: Querying a window’s password statusCComPtr<IUIAutomation> spAutomation;CComPtr<IUIAutomationElement> spElement;HRESULT hr;hr = spAutomation.CoCreateInstance(CLSID_CUIAutomation);if (SUCCEEDED(hr)){

hr = spAutomation->GetFocusedElement(&spElement);if (SUCCEEDED(hr)){

BOOL isPassword;spElement-

>get_CurrentIsPassword(&isPassword);}

}

Automation Elements

Just as in the managed API, an automation element represents a piece of UI, regardless of underlying implementation First, create an instance of

CUIAutomation, the factory for the object model

Next, get an element: From an HWND: ElementFromHandle From a Point: ElementFromPoint From current focus: GetFocusedElement

Each method returns an IUIAutomationElement

Basic Properties

IUIAutomationElement exposes properties that make sense for elements in general

Properties are exposed in COM manner and are consistent across all underlying UI

UIA Control Type

UI Framework Framework Specific

Property

UI Automati

onProperty

Button Windows Presentation Foundation

Content NameProperty

Button Win32 Caption NameProperty

Image Trident/HTML ALT NameProperty

Three Key Properties

1. NameProperty String that a user would use to explain which control

was being referred to (Customer: Accessibility Tools)

2. ControlTypeProperty Primary identifier for what type of control the

automation element represents. (Customers: Both)

3. AutomationIdProperty String that uniquely identifies an element among its

siblings in the tree (Customer: Test Automation)

Control Patterns

Control Patterns allow access to a control’s particular abilities InvokePattern:

‘Push Me’ TransformPattern:

‘Move Me’ ScrollPattern:

‘Scroll Me’ SelectionItemPattern:

‘Select Me’ Patterns can contain

properties, methods and events

ComboBox

Simple

ExpandCollaps

e

Value

Find Requests

Find requests allow you to locate elements based on conditions Conditions are based on property matching They can include Boolean operators as well

Requests can be restricted by scope Find can return the first match or all

matches Faster alternative to manual searching

Visiting the Relatives: TreeWalker

All UIA elements are organized into a tree

TreeWalkers allow navigation around the tree GetParentElement,

GetFirstChildElement,GetNextSiblingElement, etc.

Can filter by condition

List View

ListItem1 ListItem2 Scroll Bar

Up button

Down button

A sample control tree:

Consuming Events

Enable clients to be notified when changes occur in the UI Client implements a listener interface Register your listener with CUIAutomation

Clients can scope what parts of automation tree to listen for events on

Applicable events are always associated with a reference to automation element where change occurred

http://msdn.microsoft.com/en-us/library/aa359583(VS.85).aspx

15

DumpUIATree

sample

Accelerate your access: CacheRequests

Define •Create a request and add entries for properties and patterns you want cached•Define filter and scope for caching

Query •Call the element’s BuildUpdatedCache method or “…BuildCache” methods•A new copy with cached data is returned in one chunk

Use •Use ‘Cached’ flavor of property accessors to read cached data or patterns•Children can be cached as well

Revisiting DumpUIATree

sample

NEW PROPERTIES AND PATTERNS

Client ProviderUIA

Patterns for Control Virtualization

Allows controls to have more children than are exposed in Automation tree at one time Parent container permits searching for children

by property Children can be de-virtualized upon request

Useful for: Containers with massive numbers of children

Other new properties and patterns

Name PurposeARIA Properties Expose ARIA properties for

browser elements

Provider Description Query for the name of the provider(s) for an element

Synchronized Input Pattern Wait for a UI element to receive mouse/kbd input

Legacy MSAA Pattern Access original MSAA properties for a provider

Menu mode start/end events Find out when an application enters/exits menu mode

BUILDING GREAT UIA PROVIDERS

Client ProviderUIA

When do you need a provider?

You don’t always need to write a provider. Customizing beyond your framework

Changing perceived properties Annotation API may be sufficient

Adding new abilities Full custom controls

Creating new frameworks Silverlight had to implement Accessibility

Implementing the simple interface

Create a COM object that implements IRawElementProviderSimple Retrieve VARIANT properties

Implement the most important ones for your control, especially Name and ControlType

Retrieve a pattern implementation Connect your COM object to the outside

world through WM_GETOBJECT

Implementing patterns

Control patterns are much like interfacesCan reside on same

object or different object from Simple interface

Implement patterns that reflect your control’s abilities

ComboBoxSimpleExpandCollapseInvokeTableTextToggleValueVirtualized

A basic custom-control UIA provider

sample

http://msdn.microsoft.com/en-us/accessibility/cc307849.aspx

Beyond Simple Elements

Representing children Free support for mirroring HWND tree When you have non-window children …

Implement Fragment for all nodes in your tree Implement FragmentRoot for the root of your tree

Firing Events If a user would notice a change in your control,

you need to fire an event You may check for listeners first

Custom patterns and properties

Allows client and provider to exchange data beyond original Accessibility design

Custom work must be done on both sides Properties and events: business as usual Methods: extra marshalling work required

Client ProviderUIACustom dataCustom data

Custom ‘proxy’ support

Enables a client to register a substitute provider for a UI that lacks a real provider Proxy implements UIA interfaces on behalf of

UI Useful for legacy code that cannot be

modified but must be made accessible

Client ProviderUIA

Custom Proxy Non-UIA channel

WRAPPING IT UP

Is this all of Accessibility?

Is your software usable with … Programmatic access? Keyboard-only access? High contrast / high DPI? Slow reaction times?

Accessibility is not a yes/no question: it is usability for a specific audience

http://msdn.microsoft.com/accessibility

What about managed clients?

The new COM API was designed with .NET interoperability in mind Pro: Fastest performance for programmatic access Con: Imported COM interfaces are not identical with

older interfaces Con: Some Win32 controls have slight behavior

differences

UI Automation Core

UIA Managed Client API UIA Provider API UIA COM Client API

Your Client Code;TlbImport Wrapper

Managed client option #2:Use the existing managed library

We continue to support the existing System.Windows.Automation namespace Pro: No change to existing code Con: May be slower when accessing MSAA applications Con: Cannot access new properties and patterns with

current version

UI Automation Core

UIA Managed Client API UIA Provider API UIA COM Client API

Your Client Code

Call to Action

Use the new UI Automation Client API to interact with existing user interfaces

Create UI Automation providers for new custom controls and UI frameworks

Consider the Accessibility needs of your customers when designing applications

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.