Advisor: Prof. Francesco Tisato Tutor: Prof. Carla Simone September 23th 2009 Diego Bernini Ph.D. research status and thesis hypothesis Software Architectures.

Advisor: Prof. Francesco Tisato Tutor: Prof. Carla Simone

September 23th 2009

Diego BerniniPh.D. research status and thesis hypothesis

Software Architectures forResponsive Environments:

the case of Interactive Art & Design

University of Milano-Bicocca, Doctoral School of Science Ph.D. Program in Informatics, Cycle XXIV

Department of Informatics, Systems and Communication

September 23th 2009 D. Bernini, Ph.D. research status 2

Outline

• Reference scenario• Research topic, status and thesis hypothesis• State of the art• Plan of the activities• Miscellanea

Software Architecture

• Having a sound software architecture is a key aspect of ICT systems

• In different classes of ICT systems relevant high-level concepts are not turned into proper elements in their software architecture– need of proper architectural abstractions– examples: time and space in surveillance systems and Geographic

Information Systems (Micucci et al., 2006) (Micucci et al., 2009)

• This leads to several issues– intermix of conceptual, design aspects and implementation

details– strong dependency on specific technologies



Responsive environments

• Real physical spaces equipped with ICT technologies capable of sensing and responding accordingly to entities that inhabit them (Silva and Vasconselos 2007) (Pinhanez and Bobick 2003)

• Different aims– From more concrete …

• Automation of offices, shops, houses (Elrod et al., 1993)

– …to more aesthetics• Interactive Art & Design (Bullivant 2007) (Bullivant 2006)

• Various related research areas, different characterizations– From Software Engineering, Ubiquitous (Pervasive) Computing– to Human-Computer Interaction, Ambient Intelligence


..in Interactive Art

• ICT-supported interactive art installations– Art Installations: artworks where space is central

• “art that appropriates space to its own artistic ends” (Bestor 1996)

– Interactive Art installations: the viewer become an active player – ICT-supported: some interactions are achieved by ICT

• For itself: they are integrant part of the installation• For the exhibition: valorization of traditional artworks/installations

– Interactive museums and galleries

(Studio Azzurro,www.studioazzurro.com) (Rafael Lozano Hemmer,

http://www.lozano-hemmer.com)


.. in Interactive Design

• Various classes of interactive small-scale architectural & design interventions (Castle 2007) (Bullivant 2007) (Bullivant 2006)

• Interactivity: the environment enhance the user experience• Interactive Architecture & Design, urban-open & closed spaces

(Digital Pavillion,ONL [Oosterhuis_Lénárd] design office

http://www.oosterhuis.nl)

(Interactive tunnel, Vito Acconci Studio and CSAI,

(Bandini et al 2008))


High-level concepts: examples• Time

– real-time responsiveness: the interactions may require strong time constraints

– temporal input analysis and time-driven output

• Space– locating objects in multiple spaces (e.g. sensor network space vs.

environment topology)

• Multi-modality– Multiple heterogeneous sensing and responding

• Sensing: – Strict physical environmental properties

» Pressure, temperature, humidity, lighting condition, noise levels– Entity properties

» Presence, identification, motion, contact, physiological

• Responding: audio and visual effects, commanding of specific devices


Current practice

• Various development tools, but– no explicit treatment of the previous high level concepts!– low level of abstraction

• Hardware computing devices: “low level” programming– E.g. Arduino, input, output & computing device, C programming

– oriented to specific technologies• Different visual programming language & tools, but too specific

– E.g. Cycling’74 Max, oriented to audio and image processing

• … consequently, they hardly address issues like:– Management of complexity: do not add complexity to the

inherent complexity of the desiderata interactions– Integration: how to integrate new and heterogeneous

technologies – Scalability: from few homogenous sensors to a lot of

heterogeneous sensors


Research topic

Filling the gap between low-level, technology oriented aspects and high-level concepts of the previous

response environments

• What are suitable architectural abstractions for the development of responsive environments in Interactive Art & Design? (Trifonova 2008) (Edmonds et al., 2005) (Machin 2002)


What I Have Done

• Characterization of a thesis hypothesis• Investigation of the state of the art • Identification of case studies and collaborations (in

progress)

• GAS (“Grandi Attrezzature Scientifiche”) project (UniMiB-DISCo): location aware, multimodal applications – my overall collaboration: architectural abstractions for the

integration of heterogeneous components• they allow the establishment of communication flows based on

space and time

– my current specific task: design and implementation of a supporting platform (in progress)


Thesis hypothesis (what)

• Devising effective architectural abstractions for responsive environments in Interactive Art and Design that:– Goal 1: abstract from the specific technologies

(hardware-input/output and computing devices- and software- specific programming languages & components-)

– Goal 2: go beyond ad-hoc approaches, fulfillment of real-time and multimodal requirements in an effective, robust and scalable way

– Goal 3: are usable by artists & designers, not only technologists– Goal 4: are applicable to other purposes, e.g. Video-surveillance,

Domotics

• Design and development of experimental applications based on the proposed abstractions– One for Interactive Art & Design, another for a different context


Thesis hypothesis (how and who)

• Core approach: Software Architectures (Prof. Tisato) enhanced with other disciplinary perspectives– Ambient Intelligence– Interaction Design/Human-Computer Interaction– Collaborative work and Knowledge Management – Prof. Tisato, Prof. Archetti, Prof. Bandini, Prof. De Michelis, Prof.

Schettini, Prof. Simone...

• Identified case studies, collaborations and abroad period– A specific workgroup among CSAI center, DISCo-ITIS and DISCo-

SAL labs– Studio Azzurro (ref. Prof. De Michelis)– GAS project – Abroad period: acquiring knowledge & skills in interactivity vs.

Art & Design vs. ICT• e.g. The Mixed Reality Lab, University of Nottingham (ref. Prof. De

Michelis)


Physical environment

OU

T D

EVIC

ES

IN D

EVIC

ES

State of Art: a reference schema

– It does NOT imply a centralized design organization and/or deployment

ICT System

Computing & Communication

Input flow Output flowResponsive environment

ICT System organization: from (purely) reactive,

stateful/reflexive, to emergent behaviors

centralized-federated

highlydistributed

Deployment

Different conceptualmodels are adopted:from Data flow modelsto Agents


State of the Art: enabling technologies

• Input: sensors, vision & tracking systems, tangible & wearable devices– Values of strict physical environmental properties and entity

properties, a survey in (Cook and Das 2007)

– Common technologies:• Wireless sensor networks, a survey in (Yick et al., 2008)• Cameras, image detection and tracking systems

– Object tracking, human motion capture ad analysis, surveys in (Yilmaz et al., 2006) (Moeslund et al., 2006)

– Face recognition (Colombo et al., 2007) and other systems: DISCo-Imaging and Vision Lab

• Tangible interfaces and wearable devices (Harrison and Hudson 2008)

• Output: lights, projectors, displays, speakers, sound synthesizers and other specific devices– Some common protocols: DMX (lights), MIDI (sound), Power Line

Communication (PLC) and X10 (general electronic devices)


State of the Art: tools & programming languages

• Arduino– “an open-source electronics prototyping platform

based on flexible, easy-to-use hardware and software”– http://www.arduino.cc/

• Cycling ’74 Max– Visual programming language, http://www.cycling74.com– System as computational graph of objects (patches) representing

input/output devices, processing and data containers• Objects: self-contained program, they communicate by messages

– Data flow oriented model, high extensible– MSP package: audio synthesis– Jitter package: video processing– Support for execution in local networks

• Others: Pure Data (PD), vvvv, Quartz Composer– Core-model similar to MAX, but different focus


State of the Art: architectural abstractions• Software Components

– (Heineman and Council 2001), (Shaw and Garlan 1996), component styles (Garlan 1996) (Maier and Rechtin 2000)

– Specific component typologies: components with ports and connectors (Caflisch et al., 2005), (Senthil et al., 2009), targeted to responsive environments: context components (Jacquet et al., 2005), e-Gadgets (Markopoulos et al., 2004)

• Time and space awareness– time (De Paoli and Tisato 1995), (Micucci et al., 2006), space

(Micucci et al., 2009), GAS project

• Emergent conceptual and architectural models– Multi-Agent models, Cellular Automata (CA) models, suitable for

emergent and lifelike behaviors– (Bandini et al., 2008), (Silva and Vasconcelos 2007), (Sparacino et

al., 2001)


State of the art: conceptual frameworks

• Time and space– time, space and audience vs. interactivity (Benford et al., 2009)

• Multi-modality– multi-modality categories (Bongers and Veer 2007)– typologies of sensing (Benford et al., 2005)– human movement representation and responsive environments

(Loke and Robertson 2009)


Plan of Activities

• Phase 1 (investigation) - 12 months– Investigation of the state of the art (completed)– Identification of case studies & collaborations (currently in

progress)

• Phase 2 (research) – 12 months– Devising and development of the architectural abstractions, by

generalizing the analysis of the identified case studies

• Phase 3 (experiment) – 8 months– Design and development of experimental applications based on

the proposed abstractions

• Phase 4 (research) - 4 months– Refinement of the architectural abstractions on the basis of the

previous experimental applications


Wishful thinking

• How artists/designers would like to conceive the interactivity, the audience/user participation, independently from available technologies?– From collaborations with artists/designers!– This is NOT A GOAL of the thesis


Target Conferences and Journals

• Conferences (and related Symposia)– IEEE/IFIP Conference on Software Architecture (WICSA) and European

Conference on Software Architecture (ECSA)– ACM Special Interest Group on Graphics and Interactive Techniques

(SIGGRAPH) conferences• ACM’s Creativity & Cognition conferences• ACM’s International conference on Digital Interactive Media in

Entertainment and Arts – ACM Special Interest Group for Computer–Human Interaction (SIGCHI)

conferences– ACM International Conference on Multimedia– International Conference on Tangible and Embedded Interaction (TEI)

• Journals– ACM Transactions on Computer Systems (TOCS)– Pervasive and Mobile Computing, Elsevier (PMC)– ACM Transactions on Computer-Human Interaction (TOCHI)– ACM Transactions on Multimedia, Applications, and Computing

(TOMCCAP)


Courses

• Attended courses:– “Knowledge Management: conoscenza, condivisione,

tecnologia”, Prof. Simone– “Interaction Design”, Prof. De Michelis– Others

• “Laboratorio di Comunicazione” (from QUASI_SI Ph.D. Program)• “Comunicazione della Scienza” (Doctoral School of Science)• “Gestione dei progetti di ricerca” (Doctoral School of Science)• English course (DISCo), ongoing

• Next courses:– 2/3 internal courses

• “Aspetti epistemologici dell’informatica”, DISCo, Prof. Bandini• Enabling technologies: courses of Prof. Bisiani & Schettini?

– 2/1 external courses/Ph.D. schools


Other collaborations

• F.I.R.B. - Integrated Systems for Emergency project– two papers related to my Master’s degree thesis

• Bernini, D., Toscani, D., and Frigerio, M. A software architecture for the deployment of executable transformation models. Proceedings of the 2009 International Conference on Wireless Communications and Mobile Computing: Connecting the World Wirelessly, ACM (2009), 47-51.

• Toscani, D., Frigerio, M., and Bernini, D. Dynamic update of data analysis models in emergency systems. Proceedings of the 2009 International Conference on Wireless Communications and Mobile Computing: Connecting the World Wirelessly, ACM (2009), 37-41.

– workshop participations and talks• Leipzig, 21-24 June 2009, International Workshop on Advanced

Topics in Mobile Computing for Emergency Management: Communication and Computing Platforms (MCEM 2009), presentation of (Bernini et al., 2009)

• Rome, 3-4 April 2009, InSyEme internal workshop, talk “From Application Requirements to Application Architecture”


References (1)

• Overview of Responsive Environments, Interactive Art & Design– Bullivant, L. Responsive Environments: architecture, art and design. Victoria and

Albert Museum, 2006.– Bullivant, L. 4dSocial: Interactive Design Environments. John Wiley & Sons, 2007.– Castle, H. Editorial. Architectural Design 77, 4 (2007), 4-5.– Elrod, S., Hall, G., Costanza, R., Dixon, M., and Rivières, J.D. Responsive office

environments. Commun. ACM 36, 7 (1993), 84-85.– Edmonds, E.A., Weakley, A., Candy, L., Fell, M., Knott, R., and Pauletto, S. The

studio as laboratory: combining creative practice and digital technology research. Int. J. Hum.-Comput. Stud. 63, 4-5 (2005), 452-481.

– Machin, C.H.C. Digital Artworks: Bridging the Technology Gap. Proceedings of the 20th UK conference on Eurographics, IEEE Computer Society (2002), 16.

– Trifonova, A., Jaccheri, L., and Bergaust, K. Software engineering issues in interactive installation art. International Journal of Arts and Technology 1, 1 (2008), 43-65.


References (2)

• Enabling technologies, tools & programming languages– Bestor, C. MAX as an overall control mechanism for multidiscipline installation art.

Computers & Mathematics with Applications 32, 1 (1996), 11-16.– Cook, D.J. and Das, S.K. How smart are our environments? An updated look at the

state of the art. Pervasive Mob. Comput. 3, 2 (2007), 53-73. – Colombo, A., Cusano, C., and Schettini, R. Face^3 a 2D+3D Robust Face

Recognition System. Proceedings of the 14th International Conference on Image Analysis and Processing, IEEE Computer Society (2007), 393-398.

– Harrison, C. and Hudson, S.E. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. Proceedings of the 21st annual ACM symposium on User interface software and technology, ACM (2008), 205-208.

– Pinhanez, C.S. and Bobick, A.F. Interval scripts: a programming paradigm for interactive environments and agents. Personal Ubiquitous Comput. 7, 1 (2003), 1-21.

– Moeslund, T.B., Hilton, A., and Krüger, V. A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104, 2 (2006), 90-126.

– Yick, J., Mukherjee, B., and Ghosal, D. Wireless sensor network survey. Comput. Netw. 52, 12 (2008), 2292-2330.

– Yilmaz, A., Javed, O., and Shah, M. Object tracking: A survey. ACM Comput. Surv. 38, 4 (2006), 13.


References (3)

• Software architectures and architectural abstractions– Caflisch, L., Savigni, A., Schettini, R., and Tisato, F. A software architecture for real-time,

embedded monitoring systems. Advanced Video and Signal Based Surveillance, 2005. AVSS 2005. IEEE Conference on, (2005), 540-545.

– De Paoli, F. and Tisato, F. Architectural Abstractions for Real-Time Software. Proceedings of the Second Asia Pacific Software Engineering Conference, IEEE Computer Society (1995), 199.

– Heineman, G.T. e Councill, W.T., cur. Component-based software engineering: putting the pieces together. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2001.

– Garlan, D. and Shaw, M. An Introduction to Software Architecture. Carnegie Mellon University, 1994.

– Jacquet, C., Bourda, Y., and Bellik, Y. An architecture for ambient computing. Intelligent Environments, 2005. The IEE International Workshop on (Ref. No. 2005/11059), (2005), 47-54.

– Markopoulos, P., Mavrommati, I., and Kameas, A. End-User Configuration of Ambient Intelligence Environments: Feasibility from a User Perspective. In Ambient Intelligence. 2004, 243-254.

– Maier, M.W. and Rechtin, E. The art of systems architecting (2nd ed.). CRC Press, Inc., 2000. – Micucci, D., Oldani, M. and Tisato, F. Time-Aware Multi Agent Systems. In Proocedings of

Multiagent Systems and Software Architecture (MASSA). D. Weyns and T. Holvoet (Eds.), 2006, 71-78.

– Micucci, D., Tisato, F., and Adorni, M. Engineering spatial concepts. Knowl. Eng. Rev. 24, 1 (2009), 77-93.

– Senthil, R., Kushwaha, D.S., and Misra, A.K. An improved component model for component based software engineering. SIGSOFT Softw. Eng. Notes 32, 4 (2007), 9.

– Shaw, M. e Garlan, D. Software architecture: perspectives on an emerging discipline. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1996.


References (4)• Emergent conceptual and architectural models

– Bandini, S., Bonomi, A., Vizzari, G., et al. A CA-Based Approach to Self-Organized Adaptive Environments: The Case of an Illumination Facility. Proceedings of the 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops - Volume 00, IEEE Computer Society (2008), 1-6.

– Silva, F.S.C.D. e Vasconcelos, W.W. MANAGING RESPONSIVE ENVIRONMENTS WITH SOFTWARE AGENTS. Appl. Artif. Intell. 21, 4-5 (2007), 469-488.

– Sparacino, F., Davenport, G., and Pentland, A. Media in performance: interactive spaces for dance, theater, circus, and museum exhibits. IBM Syst. J. 39, 3-4 (2000), 510, 479.

• Conceptual frameworks– Benford, S., Giannachi, G., Koleva, B., and Rodden, T. From interaction to

trajectories: designing coherent journeys through user experiences. Proceedings of the 27th international conference on Human factors in computing systems, ACM (2009), 709-718.

– Benford, S., Schnädelbach, H., Koleva, B., et al. Expected, sensed, and desired: A framework for designing sensing-based interaction. ACM Trans. Comput.-Hum. Interact. 12, 1 (2005), 3-30.

– Bongers, B. and Veer, G.C. Towards a Multimodal Interaction Space: categorisation and applications. Personal Ubiquitous Comput. 11, 8 (2007), 609-619.

– Loke, L. and Robertson, T. Design representations of moving bodies for interactive, motion-sensing spaces. Int. J. Hum.-Comput. Stud. 67, 4 (2009), 394-410.

Advisor: Prof. Francesco Tisato Tutor: Prof. Carla Simone September 23th 2009 Diego Bernini Ph.D. research status and thesis hypothesis Software Architectures.

Documents

interactive art ict

research status3 slide

interactive tunnel

research status5

research status6

art plan

central art

design aspects