A multi-perspective digital library to facilitate integrating teaching research methods across the computing curriculum

A Multi-Perspective Digital Library to Facilitate IntegratingTeaching Research Methods Across the Computing

Curriculum

Anne G. ApplinDepartment of Computer Science

Ithaca [email protected]

Hilary J. HolzDepartment of Mathematics and

Computer ScienceCal State, East Bay

[email protected]

William JoelDepartment of Computer Science

Western Connecticut StateUniversity

[email protected]

Ifeyinwa OkoyeDepartment of Mathematics and

Computer ScienceCal State, East Bay

[email protected]

Katherine DeibelComputer Science and

EngineeringUniversity of Washington

[email protected]

Becky GrasserInformation Technology &

Computer ScienceLakeland Community College

[email protected]

Briony J. OatesSchool of ComputingUniversity of Teesside

[email protected]

Gwendolyne WoodSchool of Leadership andProfessional Advancement

Duquesne [email protected]

ABSTRACTThe computing research methods (CRM) literature is scat-tered across discourse communities and published in spe-cialty journals and conference proceedings. This dispersionhas led to the use of inconsistent terminology when referringto CRM. With no established CRM vocabulary and isolateddiscourse communities, computing as a field needs to engagein a sense-making process to establish the common groundnecessary to support meaningful dialog.

We propose to establish common ground throughthe construction of the computing research methodsmulti-perspective digital library (CRM-MPDL), a scholar-produced digital resource for the CRM community. As withits parent design research project on teaching CRM, CRM-MPDL is being developed through iterative and participa-tory design in an emergent fashion in tandem with the largerCRM community.

For our resource to be viable, we must carefully explorethe rich details and nuances of our stakeholder communitiesand the perspectives they bring to the sense-making pro-cess. As a discount alternative to truly having a represen-tative sample of our user population “in the room” with usthroughout the design and implementation process, we haveimplemented a development approach for CRM-MPDL us-ing personas as a means to gain insights and feedback fromthe target user communities.

For this iteration of the development process, we are con-centrating on the needs of the faculty.

In this report, we present our evolving understanding ofthe project, and seek feedback and input on several keyaspects of the theoretical and process models. We thenpresent the framework for the faculty personas, as well asan overview of some of the personas at the time the paper

was prepared, in the hopes that we can entice readers tovisit the project website to help with the ongoing audit andrefinement process. We also give an overview of the con-tent model for CRM-MPDL, which will have evolved (andmay even be available as a working prototype) by the timethis article appears in print. Finally, we conclude with acurrent status summary, and issue several specific calls forparticipation in the ongoing work of the project.

Categories and Subject DescriptorsK.3.2 [Computers and Education]: Computer andInformation Science Education—Computer science educa-tion, Information systems education, Curriculum, Literacy ;;H.5.3 [Information Interfaces and Presentation (e.g.,HCI)]: Group and Organization Interfaces

General TermsExperimentation, Standardization, Human Factors, Design

KeywordsComputing Research Methods, Sense-making,Scholar-produced Digital Resource, Personas

1. INTRODUCTIONIn March of 2005, the SIGCSE board chartered the Com-

mittee on Teaching Computer Science Research Methods(SIGCSE-CSRM) [37]. SIGCSE-CSRM is the ongoing or-ganizing structure for what has grown into an iterative andparticipatory design research project on teaching computingresearch methods. ITiCSE (Innovation and Technology inComputer Science Education) 2007 Working Group (WG) 4,

184

A Multi-perspective Digital Library to Facilitate IntegratingTeaching Research Methods Across the Computing Curricu-lum, was an integral part of the current phase of the project.

The project, on teaching CRM, was originally envisionedas a vehicle to develop a set of guidelines, standards, andmaterials for both undergraduate and introductory gradu-ate courses in computer science research methods. Over thethree years of the project, as our understanding of the envi-ronment in which we attempt to recruit and train computingresearchers has evolved, so has our vision for the role of re-search in the computing curriculum. This report presentsour vision and our design for realizing that vision, focusingon the role of the WG in both.

1.1 Establishing Common GroundIn preparation for a project working group at ITiCSE

2006, participants reviewed the literature, seeking to iden-tify CRM common to the computing disciplines [25]. Whenthe WG convened in Bologna, three months of research bysix experienced faculty from a diverse set of computing dis-ciplines, with the aid of the larger participatory design com-munity (see Section 3.1), failed to produce a set of methodsthat could be identified, labeled, or categorized in any con-sistent way. The WG members’ disciplines even viewed therole of the literature review as a research method inconsis-tently.

The 2006 WG discovered that many computing researchcommunities publish CRM papers in specialty journals andconference proceedings [25]. The publishers of these sourcesvary significantly. Publicly available indices such as the As-sociation for Computing Machinery (ACM) Guide to Com-puting Literature [1] and CiteSeer [12] often lag in picking upspecialty sources, or omit them altogether. While they aregenerally included in subscription indices such as Compen-dex [18], online access to Compendex is prohibitively expen-sive for primarily undergraduate institutions. Furthermore,computer science graduate students and, by extension, com-puting researchers make essentially no use of indices thatare not online [29]. Thus, the disciplinary CRM literatureis scattered across multiple publishers and indices, makingit extremely difficult to navigate.

The CRM literature is also scattered across discourse com-munities. Consider one of the CRM learning contexts identi-fied by the 2006 WG: studio, a small group learning contextthat includes summer undergraduate research (UR) pro-grams. Finding literature relevant to studios is extremelychallenging.1 In the ACM Guide to Computing Literature,using the search terms undergraduate and research, the fewhits relevant to the studio learning context are buried in ahuge number of results. Using undergraduate and studio doesmarginally better, but still produces a lot of noise. Usingundergraduate, experiential and research, however, produces12 “highly relevant” hits, many of which are actually highlyrelevant and about studio contexts. Changing experientialto experiential learning significantly improves the signal tonoise ratio, but loses the most relevant result about studiosfrom the previous search. The lack of any consistent vocabu-lary causes information to literally “fall through the cracks.”Falling back to undergraduate and experiential learning pro-duces an unmanageable number of results. Together, theabove example highlights the challenges in finding relevantliterature due to the difficulty of finding appropriate search

1The following searches were conducted on July 29th, 2007.

terms. While many papers have been published about stu-dio learning contexts, the authors use different terms andvocabulary to describe their work. Some terms overlap withterms used for different, unrelated purposes in other fields.Thus, even when the literature is collected into a single indexlike the Guide, finding relevant literature is still difficult.

Because the CRM literature is so widely scattered, re-searchers have not been able to form a collective CRM vo-cabulary. This lack of convergence is reflected in Glass,Ramesh, and Vessey’s (GRV) differing results for the re-search method component of their computing discipline re-search metadata set [21]. As we described in the previousWG’s report [25], the CRM taxonomy they produced is moreuseful for classifying information systems (IS) and softwareengineering (SE) papers than computer science (CS) papers.The classification system fails in that nearly three-fourths ofCS papers fall into a single category. The methodology usedby GRV to create the taxonomy is sound. The CS literatureGRV surveyed does not present a consistent vocabulary thatwould influence a taxonomy’s design, but rather an inconsis-tent terminology that fails to distinguish trends and classesof papers from each other.

All together, computing’s distributed publication venues,multiple discourse communities, and inconsistent terminol-ogy promotes the ongoing creation of isolated communities.Isolation then begets many challenges [3]. Without knowl-edge of what others have done, re-inventing the wheel be-comes inevitable. Without being able to converse with oth-ers, ideas become lost due to lack of collegial discussion,feedback, and insights. Without knowledge of how to ap-propriately use a methodology, research might be conductedpoorly with unconvincing or even untrustworthy results.

These challenges have spawned responses from withinthe computing community, however. Over the life of theteaching CRM project, we have discovered similar efforts,both current and historical, intended to address the chal-lenges and isolation of conducting and teaching comput-ing research. Some efforts have concerned computing as awhole (e.g. [32, 29]), while others focused on particular sub-disciplines (e.g., [4, 3, 19]). Some of these efforts have beenwell-received and even built upon by others, however, no oneof these projects has been able to establish the solid, com-mon ground necessary to support meaningful dialogue, e.g.,on curricular issues. The participatory design communityof the teaching CRM project encompasses many of the re-searchers from these existing efforts, and is actively seekingthe involvement of more.

The contrasting circumstances in IS and SE reveal thatcommon ground can be established through the construc-tivist process of sense-making [25]. Sense-making is a pro-cess in which a group engages in a series of actions to im-prove the community’s understanding of itself and its prac-tices. The ultimate goal is to pool and integrate individualperspectives and thoughts into a shared awareness. IS hasbeen engaged in this process for a long time, due in part toits strong tradition of researching research methods, trac-ing back to Van Horn’s 1973 paper, “Empirical Studies ofManagement Information Systems” [41]. As a result, IS hasbegun to approach some sort of consensus on terminologyand meaning for IS research methods, and thereby consol-idating previously isolated pockets of research. Moreover,the collective vocabulary allows and supports the successfulcreation of taxonomies like GRV’s [21].

185

SE is actively engaged in making sense of their researchmethods. For example, Sjøberg et al suggest the followingapproach to the challenges to integrating empirical researchmore deeply into SE:

Means to meet these challenges include (1)increased competence regarding how to applyand combine alternative empirical methods, (2)tighter links between academia and industry, (3)the development of common research agendaswith a focus on empirical methods, and (4) moreresources for empirical research. [38]

SE’s efforts have begun to yield results, as reflected in theutility of GRV’s taxonomy for classifying SE papers. Thelargest category contains less than half of the entries andonly 6 of the 19 methodology categories contained more than2% of the entries each [21]. This pattern suggests that aconsistent research vocabulary is beginning to emerge withinthe SE community.

With no established CRM vocabulary and isolated dis-course communities, computing needs to engage in sense-making. By establishing common ground, research effortswill benefit from increased access to relevant knowledge andcollegiate discussions. Just as in SE and IS before them,this sense-making process needs to take place among all themajor stakeholders: not only the research faculty most ac-tive in the research literature, but the rest of academia andindustry as well, an issue we will explore in more depth inSection 2.1.

1.2 Integrating Teaching CRM Across theComputing Curriculum

Continuing to broaden our perspective beyond computingto how researchers are trained in the sciences as a whole,we can see that many US computing departments are hav-ing difficulty engaging with the emerging interdisciplinaryconstructivist model for science teaching. Only one of thefour institutions chosen as “best case” exemplars for a ma-jor longitudinal evaluation of summer UR programs had acomputing component [26]. None of the examples, or thediscipline-specific links, in a trade journal article [8] aboutefforts to use inquiry-based learning to improve how scien-tists are trained concern computing.

Curricula at institutions adhering to the emerging model,“. . . emphasize teams of students working together and en-courage them to apply scientific principles and reasoning toreal-world problems – for example, how the hole in the ozonelayer has formed.” [8] These programs weave a research per-spective into their science curriculum from the start. Forexample, Hamilton College, in New York, starts a smallcohort in a 5-week summer program before their freshmanyear, with 10-week programs each subsequent summer, dur-ing which additional students join the program. After thefreshman year, students are encouraged to continue withtheir research during the academic year. The “pre-frosh”program infuses an atmosphere of research into the sciencecurriculum from the start (George Shields, private corre-spondence).

Comprehensive interdisciplinary experiential UR pro-grams at institutions such as Hamilton are treated as train-ing experiences, so every effort is made to accommodateall interested students. Students are encouraged to exploremultiple topics and fields through different courses. These

courses are not limited to single departments, but can in-clude collaborations with other departments, even those out-side the natural sciences. The courses are also changed toreflect an emphasis on research experience. Recognizing thathigh school science courses have already provided practicalbackground, introductory college courses instead are cen-tered around project-based learning. At the heart of thisapproach is prioritizing research as the environmental con-text in which learning is to take place.

Although the above example is only one way that otherfields have incorporated research into their curriculums, thepoint is that research has been integrated as a vital compo-nent of students’ educational experiences. It is our hope toachieve a similar integration of research methods across thecomputing curriculum through the encouragement and useof experiential and project-based learning.

Just as the lack of a common CRM vocabulary cannothelp but impede the participation of computing as a disci-pline in interdisciplinary experiential science programs, anexplicit effort to integrate CRM into the computing curricu-lum should aid the sense-making process described in theprevious section. Additionally, integration is one means toimprove recruitment, retention, and diversity of students inthe field [2, 26].

We are currently losing many of our most promising stu-dents to other disciplines that incorporate computers intotheir research practices, especially at the undergraduatelevel. These disciplines not only show students how com-puters can change world, but invite them to participate inthe process, giving those students the sense that they are“becoming scientists” [26] and can have a positive impact onsociety.

In contrast, most undergraduate computing courses teachfacts, skills, and techniques divorced from the underlying re-search. Research (and CRM) are frequently discussed onlyduring senior capstone courses or highly competitive UR ex-periences. We proclaim computing to be full of beauty, ex-citement, and awe [7], but make our students wait yearsto experience excitement or awe firsthand. It should be nosurprise that other disciplines attract more students—theyengage their students from the beginning!

A computing curriculum which integrates CRM shouldengage computing students the same way. For example, un-derlying research can be infused into introductory comput-ing courses as stories, to borrow a very successful approachfrom physics [15]. While faculty may express concern aboutintroducing more material into already overcrowded intro-ductory courses, one must engage students before one caneffectively teach any material, let alone attempt to compressfar too much material into too few courses.

Through the sense-making process, students, practition-ers, and researchers can assist teaching faculty in integrat-ing research methodologies and experiences into the lec-tures, course materials, assignments, and projects of exist-ing courses. Over time, a research component should be-come the norm for the computing student’s course experi-ence. Having students actively investigate to discover leadsto deeper understanding of course topics, improves conceptretention and engages students in their learning. Activelearning is empowering.

The question of integration is independent of that of CRMcourses [29, 32], or summer UR programs. Developing andoffering CRM courses and/or summer research programs is

186

extremely resource intensive in terms of faculty, space, andtime. Some institutions might provide one or both to theirstudents collaboratively, or through partnerships with affil-iated institutions.

Integrating teaching CRM into existing courses allows in-structors to engage students through their individual exper-tise and experiences. By embedding CRM into pre-existingcourses, students experience CRM in context and as inte-gral to the subject matter. Embedding research across thecurriculum achieves our goal of bringing research to the fore-front.

1.3 A Global Context

The techniques [inquiry based learning] appear tohave made the most inroads at liberal-arts col-leges. . . . . Progress has been slower at researchuniversities . . . [8]

There were few appreciable differences in effects[of UR] among graduates of different types ofschools or between those who began their under-graduate education at a 2-year college and thosewho began at a 4-year school. [35]

We chose to conduct this study [summer researchprograms] at four liberal arts colleges with along history of well developed UR programs be-cause findings would represent the “best case.”. . .Within the context of research universities –where the large majority of students undertakeUR – we predict higher variability in the qualityof these experiences and in the types of gains thatstudents take away from the experience. [26]

As this project has matured, and members have at-tempted to adapt locally successful solutions it has been acontinuous revelation to many of us to discover how little weknow about the actual structures of each others’ academiccareers, even when we are graduates of each other’s typesof institutions. Early on, when we brainstormed about howthe interests and concerns of CRM faculty might changeacross institutional type, we found ourselves tripping acrossassumptions:

• “At undergraduate-only institutions, instructors aregenerally less active in research and may be lessadept at navigating the already poorly structured dis-ciplinary CRM literature.”

• “Instructors with graduate students will likely want togo more in-depth about certain CRM.”

• “Some instructors will have non-traditional studentswho are taking courses in relation to their current ca-reers, meaning some CRM will of less benefit to knowabout.”

• . . .

Yet some exclusively undergraduate US institutions have su-perb libraries and emphasize literature skills as part of thearts & sciences tradition (see below). At those same institu-tions, it is not unusual for faculty to have very low teachingloads and a very active research program. Comparisons withnon-US institutions were even more challenging. What we

found was that we had no common frame of reference uponwhich to work.

Similar cognitive dissonance, examples of which appear inthe quotes at the beginning of this section, led to a pro-found revision of the Carnegie Classification of Institutionsof Higher EducationTM[10], the classification system fromwhich terms such as liberal arts college, research university,and two-year college ultimately derive. The 2005 revisionreframes the system as a a multi-dimensional analytical toolthat forms the heart of an ongoing sense-making process [30].It organizes the classifications “. . . around three fundamen-tal questions: what is taught (Undergraduate and GraduateInstructional Program classifications), who are the students(Enrollment Profile and Undergraduate Profile), and what isthe setting (Size & Setting).” [10] Using the revised CarnegieClassifications, similarities that were obscured by broad cat-egories such as liberal arts college and research universitybegin to emerge, as we can see when we compare GrinnellCollege, one of the four liberal arts colleges in the summerUR program study [26], to University of Rochester, one ofthe few research universities to extensively adopt inquiry-based learning in the sciences [8] (see Table 1.)

Recasting one of the WG’s brainstorming statements us-ing the revised Carnegie Classifications yields

• Faculty teaching in instructional programs with a pro-fessions focus can keep students engaged by emphasiz-ing CRM in the context of early adopters and otherpractice-oriented concerns.

A deeper benefit to introducing a framework such as therevised Carnegie Classifications into our research is that thedefinitions used in the classification are constructed within aglobal context. In From the Liberal to the Practical Arts inAmerican Colleges and Universities: Organizational Anal-ysis and Curricular Change, a formative document for therevised categories, Brint, et. al., explain:

The arts and sciences originated historically forthe pursuit of knowledge “for its own sake” and,simultaneously, as the educational foundation foryouths preparing to occupy positions of powerand influence in society. They include the basicfields of science and scholarship, such as chem-istry, economics, history, literature, mathemat-ics, philosophy, and political science. By con-trast, programs in [professional] fields are de-signed to educate students for jobs – in business,education, engineering, nursing, public adminis-tration, and many others.[9]

They go on to contrast these two uniquely US models witheducational systems in other parts of the industrializedworld, especially to member nations of the European Union,with which the US is in actively engaged in a sense-makingprocess concerning issues of transparency and transferabil-ity in higher education [31]. Undergraduate education inmost European countries does not include general educa-tion (a hallmark of the arts & sciences approach), muchmore closely resembling a US undergraduate instructionalprogram that is professionally focused. Notable exceptionsto this pattern are Scotland, England, and Germany, eachof which has its own rich academic tradition.

We are investigating how to exploit and extend the multi-faceted structure of the revised Carnegie Classifications to

187

Table 1: A comparison of Grinnell College and University of Rochester using the revised Carnegie Classi-fications reveals similarities between the two institutions that may contribute to their success in adoptingtechniques grounded in constructivist pedagogy for training scientists.

Classification Grinnell Rochester

Undergraduate Instructional Program Arts & sciences focus, no graduate co-existence

Arts & sciences focus, high graduatecoexistence

Graduate Instructional Program (Not applicable) Comprehensive doctoral with medi-cal/veterinary

Enrollment Profile Exclusively undergraduate four-year Majority undergraduate

Undergraduate Profile Full-time four-year, more selective,lower transfer-in

Full-time four-year, more selective,lower transfer-in

Size and Setting Small four-year, highly residential Medium four-year, highly residential

analyze those institutional and cultural contexts in such away that we can collaborate effectively across not just in-stitutional boundaries, but types of institutions, and educa-tional systems. The project found that the revised CarnegieClassifications supported meaningful comparisons within theWG’s institutions (including the UK member), with oneextension. The revised Carnegie Classifications retain theoriginal Carnegie Classifications’ tendency to view all post-baccalaureate study as intended to culminate in a doctor-ate. Thus, if an institution offers a single doctoral program,no information is given about postbaccalaureate programs.Our extended Carnegie Classifications differentiates betweengraduate instructional programs which only award MS de-grees“on the way”to a doctorate, and graduate instructionalprograms which include independent MS programs withinthe same academic department. The analysis of the WGmembers’ institutional context supported by our extendedCarnegie Classifications (see Table 2) reveals an intriguingtendency to combine the arts & sciences and professions ap-proaches at the undergraduate level, despite the differencesin institutional type.

To generalize and disseminate locally successful modelsfor inquiry-based learning, one must understand the insti-tutional and cultural context within which that learning oc-curs. Brainard [8] tells a cautionary tale about graduatestudents trained in new methods of science teaching who feltthey had to hide their activities from their research mentors!If we wish to avoid the fate of Physics, which is only justbeginning to emerge from a catastrophic collapse in enroll-ment caused in no small part by an inability to effect changein the way they trained researchers [24], we will have to findmore effective models for designing our agencies of change.

In the same way that human-computer interaction re-searchers speak of universal design, rather than specializeddesigns to meet the needs of specific disabilities, our ideal(one we freely admit is impossible to achieve) is a globaldesign, which would then be interpreted through the lensof cultural and individual experiences and needs. Explicitlyconstructing a framework to support global design is thefirst of three key areas in which the project is actively seek-ing participation at this time (see also Section 7.2.) Theexisting dialogue between US and EU education ministersand the emergence of a common vocabulary give us reasonto believe that the effort will yield useful results. All in all,the end lesson is that the community of computing facultyis composed of different people all of whom need differentsorts of information. To support and promote sense-makingand teaching CRM, this diversity of information needs must

be met.

2. APPROACHThe sense-making process IS followed to arrive at the tax-

onomy used by GRV took 30 years! We propose to acceleratethat process by building CRM-MPDL, a scholar-produceddigital resource designed to establish common ground forthe CRM community. Scholar-produced digital resources aredigital libraries that are produced by scholarly communitiesof practice (COP), rather than by “. . . institutions exter-nal to specialist communities such as commercial publish-ers, archives, or libraries . . . ” and “. . . include aggrega-tions of resources that support research, such as field-basedtopic gateways and bibliographies; products of original re-search such as novel multimedia publications and handbooktype web publications, communication fora such as preprintarchives and pure e-journals; and research tools such as non-proprietary software available via the web.” [20]

According to Sumner, “a repository or library is not de-fined by its technical characteristics, but as a specific map-ping between organizational capabilities and communityneeds.” [39] What is it, then, that accounts for the vari-ability in the viability of scholar-produced digital resources?Fry and Talja [20] show that some of that variability can beaccounted for by the intellectual and social characteristicsof the scholarly communities of practice that produce thoseresources.

In [20], Fry and Talja analyze field differences in the viabil-ity of scholar-produced digital resources using an extensionof Whitley’s theory of the intellectual and social organiza-tion of academic disciplines [42]. Whitley’s theory comparesdisciplines along two dimensions: mutual dependence andtask uncertainty. Mutual dependence includes: the extentto which progress in a field is dependent on knowledge pro-duced in other fields; the degree to which the careers of schol-ars in a field are linked; and the degree to which a field ac-cepts the assessment of other fields of external work, ratherthan developing its own criteria. Task uncertainty“. . . refersto the degree to which task outcomes and research processesare predictable, visible, and clearly related to general goals.”[20]

Whitley’s model was developed for established fields, anddoes not directly apply to an emergent field such as com-puting. A direct application of Whitley’s model would cor-rectly predict the lack of consensus on research methods,but also predict that computing would principally publishin extended print articles and books! Fry and Talja extend

188

Table 2: An analysis of the 2007 WG members as faculty stakeholders in CRM-MPDL. (a) Listed on this page are those members whoseperspective on CRM-MPDL is more purely one of faculty.

Anne Applin Hilary Holz Becky Grasser Briony Oates

Position Assistant Professor Associate Professor Professor Reader

ComputingProgram

– CS– 48 students– 7 faculty

– CS– 280 UG / 150 MS– 20 faculty

– IT & CS– 350 students– 12 faculty

– School of Computing– 3,400 full/part time students– 100 faculty

UG Program Balanced A & S / professions,some graduate coexistence

Balanced A & S / professions,some graduate coexistence

Associate’s Professions plus A & S, highgraduate coexistence

Grad Program Postbac. with A & S (otherdominant fields)

Postbac. comprehensive (Not applicable) Postbac. comprehensiveDoctoral, professions dominant

Enrollment Very high UG High UG Exclusively UG 2-year Majority UG

UG Profile Full-time 4-year, more selective,lower transfer-in

Medium full-time 4-year, inclusive Higher part-time 2-year Medium full-time, inclusive

Size & Setting Medium 4-year, highly residential Large 4-year, primarily nonresid. Medium 2-year Large, primarily residential

Education – PhD, CS Ed– MS, CS– BS, Mathematics & CS

– DSc, CS– MS, CS– BS, Mathematics & CS

– D.Eng., Interdisciplinary– MBA, – MCIS– BS, CE, – BS, Communications

– PhD, Information Systems– MS, Information Technology– BA, German Language & Lit.

Bio Aptitude testing had pointed meat CS, but it sounded boring. At33, I found myself unemployed,unskilled, and willing. From myBS, I went straight into an MSprogram with a teachingassistantship, and then to afull-time position at a communitycollege. A year later, I decided toget my doctorate in CS Ed. After12 years teaching the same 5courses, without active research, Iwas beginning to feel stagnatedand out of touch. I moved to ICbecause I needed a change.

I am a first generation academic. Ibegan pursuing my MS part-timebecause I was bored with my dayjob. Although not the first in myfamily to go to college, I am thefirst to engage in research, get aPhD, or pursue an academiccareer. As such, I am acutelyaware of the role of culturalknowledge in broadeningparticipation in computingresearch. Of interest is that myfather has also become active inacademic life, as a trustee of ourmutual undergraduate alma mater.

I came to academics late in life,having spent 20+ years inindustry. I have been teaching fulltime at LCC for 7 years, and 4additional years of part time (1 to2 courses per semester) teaching ata nearby university (BalancedA&S/prof, high grad coexistence,Postbac. comp., Doctoral STEMdominant, Majority UG, Mediumfull-time, 4-year, inclusive, Large4-year, primarily nonresidential.)

I came to computing after 10 yearsin industry as a personnelmanager. Frustrated with thetechnologists’ and vendors’ failureto understand my business needs,I decided, “If you can’t beat them,join them.” Once I began tomaster teaching, I took upresearch, concentrating on systemsanalysis and design within itssocial and political contexts. Thisled to my interest in how researchis undertaken in computing, andhow to support computingresearchers.

Role (Project Co-Director) What Ibring to the working group is ansolid understanding of formalquantitative educational researchdesigns and statistical methods. Ihave a mental model of what aresearch method ‘looks’ like. SinceI have no training or experience inpure computing research, methodsand designs will have to beexplained in terms such that evenI can understand them, whichshould be useful since explanationsand definitions need to beunderstood by novice computingresearchers.

(Project Co-Director) As Directorof the Laboratory for AdaptiveHypermedia and AssistiveTechnologies, I (and my students)play a dual role in this effort, asthis research involves my coredisciplinary interests in addition tomy computer science educationinterests. My lab has pioneered asuccessful undergraduate and MSresearch program which has grownto include many of the faculty inour department, and is nowserving as a model for otherinstitutions.

Because there is no “culture offormal research” at CCs, many ofthese topics are vague andmysterious. I joined this WG tocreate an environment which couldremove the mystery and start aresearch culture at CCs, reducingthe isolation and confusion CCfaculty feel when attendingconferences where our brethrenfrom more research orientedinstitutions hold court.CRM-MPDL will need to provideCC faculty and students a place tolearn about doing research anditems related to the “writing part”of research.

I bring to the WG knowledge ofresearch methods and underlyingphilosophical paradigms across thecomputing disciplines, and adetailed understanding of theempirical research methods usedextensively in IS research but farless in the other computingdisciplines. Much of my ownresearch has been in the ISdomain, where there has been adialogue about research methodsfor the past 30 years. I also bringmy experience teaching researchmethods to computing students,and the knowledge I gainedthrough writing a CRM textbook.

189

Table 2 [continued]. (b) These WG members belong to multiple stakeholder communities.

William Joel Katherine Deibel Ifeyinwa Okoye Gwendolyne Wood

Positions Director, Center for GraphicsResearch; Chair, CS

Research Assistant, Center forEng. Learning & Teaching; CSPhD student

Research Assistant, Lab. forAdaptive Hypermedia & Assist.Tech; CS MS student

Administrator, Assist. Assoc.Provost Acad. Affairs; Non-trad.student, Leadership & IT

UG Program Professions plus A & S, somegraduate coexistence

A & S plus professions, highgraduate coexistence

Balanced A & S / professions,some graduate coexistence

Balanced A & S / professions,some graduate coexistence

Grad Program Postbac. with A & S (educationdominant)

Postbac., professions dominantComprehensive doctoral withmedical/veterinary

Postbac. comprehensive Postbac. comprehensiveDoctoral, professions dominant

Enrollment Very high undergraduate Majority undergraduate High undergraduate Majority undergraduate

UG Profile Full-time 4-year, selective, highertransfer-in

Full-time 4-year, more selective,higher transfer-in

Medium full-time 4-year, inclusive Full-time 4-year, more selective,lower transfer-in

Size & Setting Medium 4-year, primarily resid. Large 4-year, primarily nonresid. Large 4-year, primarily nonresid. Medium 4-year, highly resid.

Education – PhD, Computer & ISc– MS, BS, Chemistry

– MS, CS– BS, CS

– BS, CS – MS, Leadership & IT, BA(accelerated, see bio)

Bio My experience and training amplysupport this WG. Since 1983, Ihave taught CS at various schoolsin the Northeast. My main areasof interest include Graphics &Animation, Computers in Society,and CS Ed. In addition, I havebeen a professional storytellersince the early 1990s. Theseinterests have helped the WG tocreate multi-dimensional andrichly textured personas.

I am nearing completion of mydoctorate at the University ofWashington. Although my PhDwill be in computer science, myresearch is truly interdisciplinaryand touches upon areas of literacy,educational technology, andaccessibility. Upon completion, itis my goal to become faculty in aneducation department at aresearch-focused university.

I was originally a Biochem major,but switched to CS after takingCS1. I love designing algorithms.I’m fascinated by researchinvolving intelligent systems,pattern recognition, the brain andbehavior. My goal is to get a PhDin Intelligent systems, become afaculty at a research university,mentor non-traditional researchersand do research combining CS,psychology & medicine.

I am a full time non-traditionalstudent enrolled in an acceleratedBS/MS program at DuquesneUniversity. In my program atDuquesne, which is thesis-drivenand self-defined, I am focusing onthe academic administrator’sperspective on this project.

Role I am a member of theACM/SIGGRAPH EducationCommittee, within which I headthe Undergraduate ComputerGraphics Research Initiative. Thisproject began in 2003 when Ihosted the first of 4Birds-of-a-Feather sessions at theannual SIGGRAPH conference.These BOFs produced a set ofrecommendations regardingundergraduate CG research.In 2007, I facilitated a forum atSIGGRAPH on CRM-MPDL togather opinions from the largereducational community, as well asa BOF on creatinginter-institutional partnerships.I found myself approaching thework of this project principallyfrom the role of a liaison from aresearch COP and an academicadministrator.

As a PhD student studyingresearch methods, not only do Ihave a broad knowledge base todraw upon, I also have directexperiences in how researchmethods can be taught. In mymost effective research courses andreadings, teachers and authorshave drawn upon real-worldexamples. For example, myexperimental design instructorused examples from motivationresearch. For CRM-MPDL tosupport teaching CRM, justhaving articles describing themethods might not be enough. Itseems prudent to support thecollection with examples of themethods in practice, ideallycomputing practice. I am aneducator and education researcherat heart. Identifying ways tobetter support learning is what mylife is all about.

I play a number of roles in thisproject. As a teaching assistant, Ihave found working on the facultyperspectives fascinating. I love toprogram; CRM-MPDL lets mecombine my research and technicalinterests. I was the one whorecommended Greenstone3 as ourdigital library server. Currently, Iam buried in metadataimplementation issues. Finally, asa Nigerian female CS researchstudent, I bring my personalexperiences to the upcomingstudent perspectives design cycle.I plan to collaborate with KateDeibel on that effort.

Before returning to school fulltime, I was Business Manager andAssistant to the Associate Provostof Academic Affairs at CarnegieMellon University. At CMU , Imentored undergraduate andgraduate students in managingtheir careers and their research,ran support programs for womenand minorities in science andengineering, organized STEMconferences and advised studentsthrough many different scholarshipapplication processes. I developedthe expertise to write a number ofthe items listed in the InformationNeeds of my students and my WGcolleagues. For example, I will bein charge of writing the field guideto research careers as well as anumber of the best practiceguidelines related to fundraising.

190

the model to account for the emergent and evolving aspectsof digital publishing, for example, incorporating audiencecharacteristics (is there a significant lay audience for thework?) and uncertainty over allocation of credit for work(how does one handle truly interdisciplinary work?).

Fry and Talja’s work is attractive as a model, in part, be-cause it explains so much of what we have already discoveredin our prior work. Fry and Talja’s model, for example, pre-dict the differences between focus on CRM in the literatureby COP (see Table 1 in [25]), as computing research com-munities with higher mutual dependence on other scientificfields tend to have more organized and visible taxonomies ofresearch methods.

Fry and Talja’s model has not been previously applied toany of the computing disciplines. As their research is partof their own field’s effort to develop a theory for the disci-plinary differences in the shaping of networked resources, wewill need to carefully consider and explore the rich detailsand nuances of our specific stakeholder communities. Thesecommunities will include both communities that we will helpbuild as well as existing communities we will join.

2.1 Building a CommunityWhile there are many stakeholders in this process, we will

consider here the primary ones: disciplinary COP such asACM Special Interest Groups (SIGs), computing faculty,students, academic administrators, and computing practi-tioners (see Figure 1.)

The Community of Practice that gave birth to theteaching CRM project is CS Education (CS Ed) research.The core collaborators are a CS Ed researcher and an Adap-tive Hypermedia researcher who specializes in CS Ed (seeTable 2.) The need for the project extended far beyond thisnarrow COP; it was simply within it that the right com-bination of skills to start the project coincided. Since itsinception, through iterative and participatory design (seeSection 3), the project has grown to include an ever broad-ening set of computing COP. Because the degree of mutualdependence varies widely by COP, the information needs ofcomputing researchers, practitioners, and educators will alsovary by COP. Participants with interests that bridge multi-ple COP will be instrumental in designing CRM-MPDL toserve the needs of as wide a range of computing COP as pos-sible, drawing members of those COP into the conversation,and, in time, establishing common ground.

Computing faculty have diverse information needs, andthus will have varying expectations of CRM-MPDL. A sin-gle faculty member may have diverse needs when interactingwith CRM-MPDL in different roles. For example, while be-ing fully versed in the CRM of his own specialty, he mightstill need guidance on how to teach a CRM course, as know-ing how to do something is not the same thing as knowinghow to teach it. If he were changing his area of specialty, hemight also by looking for new collaborators, or for other re-searchers with whom to discuss nuances of technique. Mean-while, he might choose to contribute information on methodsin his old area of specialty as a means of advancing his careerwhile he changes specialty area.

Both teaching and research oriented faculty are importantto CRM-MPDL. We anticipate that, at least initially, teach-ing faculty will tend to serve as the gateway for secondaryand undergraduate students, while graduate students andpostdocs will tend to serve as the gateway for research fac-

ulty (for more on this dynamic, see Section 5.1.) Thus,although many teaching faculty might think that they donot have much to contribute to CRM-MPDL, they do intheir role as mentors to potential computing researchers. Asthese faculty glean knowledge on how best to prepare theirstudents for research, they can share information throughCRM-MPDL. In turn, those research faculty that followedup with the same students can report back on the efficacy ofthe training. Research faculty can benefit significantly fromCRM-MPDL, in terms of the efficiency and viability of theirresearch labs. By ensuring that any established patterns forconducting research in their specialty(s) are accurately rep-resented in CRM-MPDL (whether through the appropriateSIG or individually), research faculty can promote their areaof specialty to those students most interested in computingresearch careers. One of this year’s WG members is a liaisonwith SIGGRAPH; the project is actively seeking represen-tatives from other COP.

Students, from secondary school through doctoral de-gree candidates in computing, are the potential next gen-eration of computing researchers. To retain those studentsthrough the transition to computing researchers, they musthave research experiences that qualify as legitimate periph-eral participation (LPP), [28] that is, participation that oc-curs within the context of a research COP and is legitimateto both the student and that research COP. CRM-MPDL isinstrumental in this process in several ways. First, the pro-cess of establishing common ground will establish visible setsof requisite knowledge for participating computing COP, acritical component in establishing the legitimacy of LPP ex-periences. Second, the design of CRM-MPDL is intended tohelp computing COP design LPP experiences for studentsat all levels of expertise, similar to successful existing pro-grams in many other sciences. Third, not only do we expectthat participation will vary across the CRM-MPDL studentcommunity, but we also expect that the student communitywill interact. For example, an undergraduate student mightcontribute by commenting about a research method they areusing in a collaborative project, a graduate student mightdiscuss methods they are considering using for a thesis, anda doctoral candidate might compare and contrast a set ofmethods that they have used, highlighting those methodswhich were particularly challenging to master or less rele-vant to their specialty. Three of this year’s WG members arestudents; we expect that students will access CRM-MPDLdirectly, in advance of curricular reforms, based on studentfeedback over the three years of the teaching CRM project.CRM-MPDL will give students a voice in the CRM com-munity. That voice should help retain more of those mostinclined towards research until they become computing fac-ulty, teaching a set of progressively more stable CRM.

Academic administrators fall into two categories whenviewed from the context of CRM-MPDL. The first cate-gory consists of dual-role academic administrators: Assis-tant Deans, Deans and Department Heads. These admin-istrators often: have experience as teaching faculty; are ei-ther elected or nominated to their positions by the faculty;and/or continue to be active as teaching faculty in additionto their administrative duties. In their role as faculty, theneeds and contributions of dual-role administrators may besimilar to those of pure computing faculty, however, we ex-pect the process of transition from faculty to administratorand the dual role itself to bring important new insights to

191

C R M � M P D L S e n s e � m a k i n gT r a n s f e r a b l e S k i l l sC u r r i c u l a rS t a n d a r d s L e a r n i n gO b j e c t sL i t e r a t u r eR e v i e w

a c a d e m i ca d m i n i s t r a t o r s s t u d e n t s

c o m p u t i n gp r a c t i t i o n e r s

c o m p u t i n gf a c u l t y

c o m p u t i n gc o m m u n i t i e so f p r a c t i c e

Figure 1: Building CRM-MPDL so that it meets the related contextualized information needs of the majorstakeholder communities will increase mutual dependence for users from all of these communities, one keyfactor in the viability of a scholar-produced digital resource. [20]

those needs and contributions. As administrators, however,they may need information such as: best practices for aprogram based on other university programs; departmentalmetrics at other universities; how to navigate collaborativegrant applications between their institution and other aca-demic institutions and/or with industry; and so on.

The second category consists of pure university adminis-trators such as Departmental or University Business Man-agers, Vice Provosts, Provosts, Vice Presidents and Presi-dents. We expect that these administrators will relate toand use CRM-MPDL very differently. They could be seek-ing: information on other institutions’ metrics such as theeffect of UR on student recruitment and retention; some-where to refer new faculty seeking current dialogues on in-terdisciplinary research; best practices for new programs andgrant applications; or guidance on how to navigate the visaprocess for incoming faculty, students and staff. The in-put of pure administrators to CRM-MPDL will be broaderand more general in scope, as well as more detail-specific forcommon subjects such as an Institution’s grant applicationprocesses.

As a tool to facilitate dialogue between faculty and dual-role and pure administrators, CRM-MPDL will save timeand be beneficial to all concerned. As students join in the

dialogue, they will engage with the information in CRM-MPDL, leading to more informed choices, not only for thestudents, but also for faculty, administrators, and industry.Establishing common ground for all is an important steptowards consensus on training researchers in computing, re-claiming many of the talented young people who have fledcomputing research and, all too often, computing itself.

Computing practitioners in research and development,whether in industrial research laboratories or commercialconcerns, are another critical class of stakeholders in CRM-MPDL. We have already discovered that we need to assuremembers of industry that we are equally interested in allforms of computing research. While both the teaching CRMproject and CRM-MPDL were born in an academic COP, forthem to thrive, we must find a way to work with all the ma-jor stakeholders [20]. Fortunately, integrating CRM acrossthe computing curriculum offers significant value to indus-try as it teaches practical skills at all stages of developmentas a computing professional.

2.2 A Shared ResourceThe information needs of CRM-MPDL’s users will vary

significantly by stakeholder community; however, the sense-making process requires a shared resource. Major concerns

192

we have identified for CRM-MPDL’s success include: pro-viding value; designing for change; a common experience;usability; and shared ownership.

To ensure that we provide real value for users from eachstakeholder community (i.e., meet real information needsfor users from that stakeholder community) we plan projectdesign cycles for each major stakeholder community. In thisdesign cycle, we are concentrating on faculty.

In addition to providing value in the present, our designfor CRM-MPDL must incorporate how the introduction ofCRM-MPDL will change the behavior of the stakeholdercommunities to ensure ongoing viability of the resource. InSection 3, we discuss our use of iterative and participatorydesign to design CRM-MPDL in an emergent fashion.

To provide a common experience of the resource whilemeeting usability standards for a wide variety of informationneeds, all users will see a common interface, but that inter-face will be customizable and, at the user’s option, adapt-able.2

For a scholar-produced digital resource to thrive, the usersof the resource must really “own” it [20]. As CRM-MPDLis built and tested, the stakeholder communities will deter-mine how to administer the resource. For example, volun-teers from the CRM-MPDL user community would review,or send out for review, materials submitted for inclusion.

2.3 Joining a CommunityA key existing community we are joining is the developer

community for Greenstone [17], a well-established multilin-gual, multiplatform and highly configurable digital libraryserver. Greenstone has a large user base that includes vari-ous government agencies and prominent universities such asUNESCO and Oxford University and also has a dedicatedteam of developers and a vibrant user community. Impor-tantly, Greenstone embodies what we consider to be a gooddevelopment platform. It is open source and easy to extendand customize. It can also be used in its entirety as thewhole library system or used simply as the backend. Withan established server like Greenstone, we can leave the coresystem development to the Greenstone developers and con-centrate on customizing the system and designing our userinterface.

In particular, we will be using Greenstone3, the new ma-jor release of the Greenstone library server, because it im-proves on Greenstone2’s flexibility, modularity and extensi-bility [17]. Greenstone3 is structured as a network of inde-pendent modules written in Java that communicate usingXML. Although Greenstone3 provides many of the featuresrequired for our underlying system, we will still need to cus-tomize the system, including adding sense-making tools andadding plug-ins to support more document formats. The useof the XML interfaces is key, as it frees us to use a variety ofcompatible agile programming languages. Agile tools, suchas test-driven development with code coverage, are as crit-ical in research as in industry: one must first verify beforeone can validate. From an interaction perspective, Green-stone has plug-ins that converts documents from different

2CRM-MPDL’s front door is always the same, but, as in theoriginal Colossal Cavern game, “Exiting a room to the northdoes not mean that you enter the next room from the south.”With adaptation enabled in CRM-MPDL, where you go andwhat things look like when you do depends on how you gotthere.

formats to XHTML. This feature ensures that the only soft-ware our users need to view a document in the library is aweb browser.

Developing a software system from the ground up takesa lot of time and design effort. Agile software development,our choice of software methodology, encourages software de-velopment from a reusability and test-driven perspective.And so in the spirit of agile software development, we havedecided to use an established open source digital libraryserver software as the platform on which we will constructCRM-MPDL.

3. DESIGNAs with the parent design research project on teaching

CRM [25], our approach to the design and development ofCRM-MPDL is based on iterative and participatory design[5]. We are designing CRM-MPDL in an emergent fashion[40] in tandem with the larger CRM community.

3.1 The SIGCSE Committee and ITiCSEWorking Groups

To date, the participatory design community has oper-ated primarily through two SIGCSE structures: a SIGCSECommittee, SIGCSE-CSRM, and a series of ITiCSE WGs.

The SIGCSE-CSRM committee listserv was establishedin 2005 in order to begin a dialogue about teaching CRM.The listserv has been, and continues to be, a rich source ofinformation for the committee. This year, in preparationfor the ITiCSE 2007 conference, the WG held initial con-versations using the listserv. Members of the listserv werequite active in the planning stages, and the WG was ableto accomplish a significant amount of work before ITiCSE,arriving at the WG session with initial outlines for the con-tent of the resource as well as interface and methodologysuggestions.

The WG members represent a wide range of academictypes. For an analysis of the 2007 WG members as fac-ulty stakeholders in CRM-MPDL, see Table 2. In addition,several members represented multiple stakeholder commu-nities. The WG included a master’s student, a doctoralstudent, and a seasoned academic administrator deeply in-volved with a well established UR program at an researchuniversity. Currently she is a full-time non-traditional stu-dent and views the CRM-MPDL from both administrativeand academic perspectives. Additionally, having a UK fac-ulty member and a Nigerian student in the group facilitatedthe group’s awareness of international diversity, althoughthis year’s WG lacked the global flavor of the 2006 WG. Amore complete discussion of each WG member’s contextu-alized information needs can be found at [37].

As previously discussed, the project is currently concen-trating on the faculty perspectives on CRM-MPDL. As partof the participatory design process, the 2007 WG developedan initial set of persona in collaboration with the ITiCSE2007 attendees (see the following two sections.)

For the next design cycle (and possibly ITiCSE 2008), aCRM-MPDL WG has started to form, composed of and di-rected by students. The student WG tells us they want tofocus on developing the student (undergraduate, postgradu-ate, and maybe even K-12) perspective of CRM-MPDL us-ing both personal experiences of its members and possiblydeveloping a set of personas based on students and their in-teractions with CRM-MPDL. Included in the students’ WG

193

Figure 2: As part of the iterative and participatory design process, we are using an interactive poster togather feedback on the personas and generate more CRM community involvement in the project. TheITiCSE 2007 WG created the initial version of the poster, shown here, during the conference itself. Theposter was therefore only on display for the last day of the conference. Despite its brief exposure, the postergenerated several substantive suggestions, including both a new persona and a major new content elementthat the project has incorporated into the design. We are revising the methodology of the interactive posterwith use.

194

would be generating a broader understanding of students’information needs from CRM-MPDL.

3.2 Contextualizing Information NeedsIn order to understand what would cause someone to par-

ticipate (contribute, search, edit, volunteer, comment) inCRM-MPDL, we must first understand the perspective theyhave constructed from which they interact with the system[16]. Kuhlthau states, “An information search is a processof construction which involves the whole experience of theperson, feelings as well as thoughts and actions.” [27]

Ideally, participatory design [36] alone would avoid thesituation of just designing for our own needs. However, itis difficult, if not impossible, to have ready access to usersfrom all our stakeholder communities at our convenience[22]. Therefore, we are going to use an additional tool asproxy for our participatory design community: personas.

3.3 PersonasPersonas [13, 23] are a design technique that creates fic-

titious characters as stand-ins for real users of the systembeing designed. A persona describes a potential user andincludes a list of what that user will require of, or needfrom, the system being designed. Ideally, a persona rep-resents a realistic user and is supported by data collectedfrom demographics, user interviews, and other informationsources [34].3 During the design process, designers and de-velopers refer to personas as real people and mention themin design discussions and decisions. To aid this interaction,personas are given names and personal background, therebyincreasing their realism and memorability. Ultimately, a setof personas can provide a shared focus and vocabulary fordesign conversations.

An example of a persona created for CRM-MPDL isMikael, a first-year faculty member at Ivy Covered College(ICC), a private not-for-profit college with an arts & sciencesundergraduate instructional program. Mikael completed hisPhD last year with a thesis on adaptive user interfaces andis starting his first year as faculty at ICC. Although he lovescomputer science research and academia, he has grown tiredof the graduate process and is more interested in teachingand continuing his research. As ICC has only undergradu-ates and the university president has challenged the facultyto engage them in more research, Mikael is quite excited andmotivated to do user interface research with his students.However, he has no experience doing research with under-graduates, nor does he know how to start such a researchprogram.

Mikael is one of the eighteen personas we have designedto date to capture one of the multiple faculty perspectivesrelevant to the design of CRM-MPDL. These personas re-flect different types of positions such as first-time faculty,pre-tenure, full professors, etc. They also consider differenttypes of academic institutions. The personas work at one ofeight fictional institutions providing broad coverage of theextended Carnegie Classifications model discussed in Section1.3. In total, the personas developed thus far reveal that theCRM information needs of just the faculty are widely vary-ing and diverse. Given the large number of personas that we

3Our personas are based on reality, with two exceptions:gender and ethnicity. We consciously decided that our per-sonas should reflect a greater diversity than currently foundin the computing faculty population.

have created, we recognize the danger of reducing them totheir information needs. To maintain the richness of the per-sonas, we have given them a social structure by embeddingthem within an institutional context.

Because the 2007 WG members were primarily from USinstitutions, most of the personas developed by the WG rep-resent US faculty, the exception being Philip, a reader at aBritish university. Realizing the narrowness of the set, theWG approached several international project members atITiCSE (see Section 8), and in keeping with a global de-sign approach, started work on Ronit, an Israeli high schoolteacher. Ronit is very authentic, reflecting the high level ofparticipation of the Israeli computing community in K-12computing as well as providing a K-12 persona and an Is-raeli persona, meeting a variety of needs. The idea for Ronitcame from the international project members, not from theWG. The authenticity of the voices of the personas dependson just this sort of participation (see Call for Participationin Section 7.2.)

To assist in auditing the personas, during ITiCSE 2007, weposted a collection of the completed personas in the posterarea (see Figure 2.) Conference attendees were invited toread the personas to understand the goals of CRM-MPDL aswell as to provide feedback. Although the display was onlyup for a few hours, a number of substantive suggestions weremade. The feedback included suggested revisions to makethe personas more realistic as well as the creation of a newpersona that will be included in the project. Suggestionswere made about the content of CRM-MPDL, as well as thepersonas.

We will continue to audit and refine the personas throughinteractive displays at conference in multiple computingCOP and through the project website.

4. THE 2006 FRAMEWORKThe lack of a common CRM vocabulary makes providing

computing students with a general understanding of CRMvery challenging. While the sense-making process that thisproject has set in motion is intended to address that problemover time, we still need to teach today’s students! To meetthe immediate needs of their current students and fellowteaching faculty, the 2006 WG developed a broad frameworkfor teaching CRM during this sense-making process (see Fig-ure 3) [25]. In [25], the framework was verified against sev-eral master’s and doctoral level research projects and foundto describe each in a general way. To illustrate and auditthe framework, we apply it to the prior, current, and nextcycle of our project.

Prior Cycle (2006)

A What do we want to achieve?Understand the teaching of research methods in com-puting.

B Where does the data come from?Read (the literature). Brainstorm within the WG atITiCSE 2006.

C What do we do with the data?Create a framework/taxonomy (i.e. Figure 3 plus var-ious tables/figures in 2006 report.)

D Have we achieved our goal?For the group - probably yes. For the wider commu-

195

W h a t d o w e w a n t t o a c h i e v e ?F i n d o u t w h a t i s h a p p e n i n gD e v e l o p s o m e t h i n g t h a t w o r k sE v a l u a t e a n e x i s t i n g s y s t e m / t e c h n o l o g yC o m p a r e e x i s t i n g s y s t e m sC h a n g e h u m a n b e h a v i o rH a v e w e a c h i e v e d o u r g o a l ?D r a w c o n c l u s i o n sE v a l u a t e r e s u l t sI d e n t i f y l i m i t a t i o n s W h a t d o w e d o w i t h t h e d a t a ?I d e n t i f y t h e m e s / p a t t e r n s / q u o t e sC a l c u l a t e n u m b e r sI d e n t i f y t r e n d sE x p r e s s v i a m u l t i m e d i aC r e a t e f r a m e w o r k s / t a x o n o m i e s

W h e r e d o e s t h e d a t a c o m e f r o m ?R e a dO b s e r v eA s kM e a s u r eE x p e r i m e n tM o d e l F i e l dL a b o r a t o r yC o n c e p t u a lH o w t o c o l l e c t ? W h e r e t o c o l l e c t ?

Figure 3: (From [25]) A framework designed to facilitate teaching CRM during sense-making process.

nity - still to be decided. What feedback has therebeen? Limitation: The framework and taxonomies arebased only on the perceptions and experience of theWG members. Solution: widen the conversation byinvolving the wider computing community. One wayof doing this is via a scholar-produced digital resource.

One important finding emerging from the WG re-search: the CRM literature is widely scattered andoften hard or impossible to access for faculty at insti-tutions that emphasize the development of curricularmaterials. A scholar-produced digital resource couldalso help with this finding, as well.

Current Cycle (2007 – 2008)

A What do we want to achieve?Develop something that works, i.e., build a scholar-produced digital resource and associated tools. Theresource will initially be designed from the perspectiveof the faculty member that will use the information.Other perspectives will come later.

B Where does the data come from?Ask - use the personas created by the WG (see Section3.3 for explanation of personas) and the WG members,SIGCSE-CRM listserv members and CRM communitymembers at large to provide suggested references orfull-text papers, teaching resources etc. Read - butwhat literature? Relevant COP include: digital li-brary, information science, interaction design research,education design research, science education. And, ofcourse, CS Ed. Model - develop a draft knowledgerepresentation schema for the data and metadata ofthe digital resource, and a specification of the func-tions/tools to be provided.

C What do we do with the data?Create a global framework for understanding how ourteaching contexts relate to each other. Express viamultimedia, both in CRM-MPDL and the reports, pre-sentations, etc., in the participatory design process.Use to shape how we build the tool.

D Have we achieved our goal?Layered evaluation. We need to evaluate whether we

meet each sub-goal, because we build on our interme-diate results. Audit the personas, prototype the inter-action design, etc.

An important finding emerging from the 2007 WG’s workwas that the framework for making sense of computing re-search (see Figure 3) needed amending: a further dimen-sion needs to be added, asking questions about the researchethics and the human subjects and participants as we dis-cuss in Section 5.3.

Next Cycle (2008 and onwards)

A What do we want to achieve? Develop something thatworks - create meaningful dialogue and sense-makingfor teaching CRM. Augment CRM-MPDL with otherstakeholder perspectives.

B Where does the data come from?Ask - ask people to contribute to CRM-MPDL, mak-ing their sense-making available to a wider community.Observe - who contributes to CRM-MPDL, and whatis contributed. Read - what people contribute, plus anongoing reading of the literature for further resourcesto include in CRM-MPDL.

C What do we do with the data?Identify themes, patterns, quotes, trends, gaps, andusage statistics. Have a sense-making dialogue withinthe community.

D Have we achieved our goal?Not yet. We need to think about how we will knowif/when we have achieved it. For example: feedbackfrom users of CRM-MPDL regarding strengths, weak-nesses and omissions. Incorporate a discussion forum?Or use SIGCSE-CRM listserv? Look for evidence ofan emerging shared understanding of CRM. Feedbackfrom the wider community - do they know about CRM-MPDL, why they do or don’t use it.

Once CRM-MPDL is ready to go live, a user protocol willneed to be devised, and each participant should agree to itsterms. The protocol will include giving informed consentnoting that online contributions will be observed in order

196

that the researchers can monitor how CRM-MPDL growsand is used, and the observations will be based on the guide-lines for Internet research produced by The Association ofInternet Researchers [www.aoir.org]. Since this phase of theresearch will use data from human subjects, it must be ap-proved by a human subjects review board.

The initial interface testing will involve volunteers fromthe faculty and students at CSUEB. After using CRM-MPDL, each volunteer will be interviewed about their ex-perience. These interviews will be taped and qualitativelyanalysed. Additional interaction testing will include bothquantitative and qualitative analyses of user experiences. Ahuman subjects protocol for the initial phase of the researchhas been approved by the Institutional Review Board atCSUEB. The ethical aspects of the research are discussedfurther in Section 5.3.

5. RESULTS

5.1 Information NeedsIn order to fulfill our users’ information needs, we have to

understand what produces these needs. We have to under-stand what leads an individual to use our system:

• The qualities of our user’s need

• The motivations behind the need

• The type of institution where the need will be ad-dressed

• The time frame for addressing the need

• Etc.

We are creating a system that grounds users’ needs in con-text, thereby allowing the qualities of their needs to be ex-pressed. Because we must understand circumstances sur-rounding those needs, we cannot simply extract our per-sonas’ needs.

For example, consider the personas in Table 3. Alex atResearch ’R Us (R’RU) and Rachelle at Enormous StateUniversity (ESU) have expressed similar needs to acquirefunding for their labs. The parameters for these needs dif-fer. Rachelle’s department requires her to obtain half ofher funding from the National Science Foundation (NSF),while Alex does not have this constraint. NSF is a presti-gious government funding agency; grants from NSF conferexternal recognition as well as bring substantial additionalfunding to the institution itself in the form of overhead. Ap-plying for, and getting a NSF grant requires the applicantto have in-depth knowledge of the entire process, time andcommitment on the part of the applicant, and knowledge ofinstitutional practices and procedures for submitting grants.Although Rachelle’s and Alex’s need to acquire funding isessentially the same, they are not because of Rachelle’s fund-ing acquisition constraint. Rachelle has to understand theprocess of becoming a strong NSF grant applicant as shecannot pursue other funding opportunities open to Alex.

As another example, Tricia at Matriculation CommunityCollege (MCC) and Alex have expressed a need to start aresearch lab. However, their needs, and how they will ful-fill them, differ because the academic institutions involvedare so different. Tricia has to convince MCC’s policy mak-ers that establishing a research lab and teaching CRM will

benefit the students and thereby, the institution. Alex facesdifferent issues at R’RU. Her school already encourages allprofessors to establish a research lab. What is not readilyapparent in Alex’s similar need is that she is up for tenure.As she has a limited amount of time in which to accomplishthis task, she is desperate and needs information in shortorder. Tricia, however, is just exploring the idea of settingup a lab. Moreover, Tricia’s tenure is not based on settingup a lab, as she can qualify for tenure based on her teachingalone. Nothing is at stake for her, and she is under no timecontraints. That said, however, it is Tricia’s personal goalto engage her students in research so learning about a labsetup is more about personal and professional enrichmentthan job expectations as in Alex’s case. Although Tricia’sand Alex’s need to establish a research lab appears to beidentical, the factors surrounding the need create differentinvestments from CRM-MPDL users.

The project conducted a full analysis of the contextualizedinformation needs of the 2007 WG members, as well as oneach of the project personas. These analyses are availableon the project website [37]. We invite contextualized infor-mation needs from the community, not necessarily limitedto the faculty perspective.

5.2 CRM-MPDL ContentTable 4 presents a snapshot of the planned content for

CRM-MPDL. Within CRM-MPDL, content is not basedon or organized according to a hierarchy, but on how thecontent is used in context. As a scholar-produced digitalresource, all content will be reviewed prior to inclusion inCRM-MPDL. Full-text versions of artifacts will be includedwhen possible. Artifacts will include, but not be limited to,papers, books, proposals, reports, databases, metrics, etc.As a participatory design research project, we welcome in-put on CRM-MPDL content from the larger community.

Alternative views are also planned. For example, in atheme-based view, one could request all best practices guide-lines.

The design of CRM-MPDL blurs the line between whatwould traditionally be thought of as “content” vs. “tools” toimprove the interactivity within the community of practice(see Table 4.) Interaction is key to the sense-making pro-cess, because under the constructivist model, knowledge iscreated in the process of scientific communication. Thus, forCRM-MPDL to accelerate the process of building commonground, we need to emphasize the sense-making model forscientific communication.4

In addition to the tools listed in Table 4, CRM-MPDLwill have a set of sense-making tools accessible throughoutthe library. Two such tools are currently planned: a textannotation tool and a graphical tool to portray relationshipsamong items in CRM-MPDL.

Visitors to CRM-MPDL can view any materials anony-mously. Using the tools will require registration. Our intentis to minimize barriers to registration, e.g., using a databaseof university domain names to prefill university informationwhen users provide university email addresses and allowingusers the option of anonymous registration, in which theyfill in their information by institutional profile without pro-viding the actual name of the institution.

4Patterns of participation within the participatory designcommunity is an area of interest of some project members,and the subject of a paper currently in progress.

197

Table 3: An overview of some of the personas developed for CRM-MPDL. For full information on all of theproject personas, visit http://acc.csueastbay.edu/∼csrm/.

School Persona Information Needs

Matriculation Community College– Associate’s– Exclusively undergraduate two-year– Higher part-time two-year– Very large two-year– Department: 350 students / 12 faculty

Tricia (37; PhD)– Tenure-track, Assistant Professor– Teaches at only college in town– Husband (PhD) manages

computing research facility intown.

– How to introduce students toresearch

– How to start a research lab

Ivy Covered College– Arts & sciences focus, no graduate

coexistence– Exclusively undergraduate four-year– Full-time four-year, more selective, lower

transfer-in– Small four-year, highly residential– Department: 18 students / 4 faculty– University president recently challenged

faculty to do more research andemphasized importance of UR.

Mikael (24; PhD)– Tenure-track, Assistant Professor– Loves user interface toys

Onkar (53; PhD)– Tenured, Full professor– Department chair– Not happy at push for UR– Knows CS at ICC currently lacks

direction and identity

– How to start an UR program– How to supervise UR

– How to help faculty update andmodernize their class

– How to find funding for upkeep oflabs and libraries

Research ’R Us– Arts & sciences focus plus professions,

high graduate coexistence– Comprehensive doctoral with

medical/veterinary– Majority undergraduate– Full-time four-year, more selective, lower

transfer-in– Large four-year, highly residential– Department: 300 UG / 50 PhD / 15

faculty

Alex(andra) (34; PhD)– Tenure-track, Assistant professor– Used up start-up funds and release

time from courses– Sole database researcher at R’RU

– How to establish a research lab– How to recruit students– How to find funding for the lab

Philanthropic University– Professions focus, some graduate

coexistence– Postbaccalaureate professional (business

dominant)– Majority undergraduate– Full-time four-year, inclusive– Small four-year, primarily nonresidential– Non-tenure institution– Major distance-learning online presence

Yasha (45, PhD)– Fixed-term, renewable contract– Motivated to teach students the

best he can, stay current andcontinue rolling contract

– How to introduce research skills inhis class

– How to organize collaborativeresearch projects forundergraduate students

Enormous State University– Balanced arts & sciences/professions,

high graduate coexistence– Comprehensive doctoral (no

medical/veterinary)– Majority undergraduate– Full-time four-year, more selective,

higher transfer-in– Large four-year, primarily nonresidential– Department size: 500 undergraduate /

200 graduate / 30 faculty

Rachelle (32; PhD)– Tenure-track Assistant professor– No support or peer network– Teaches CS2 and Intro to Web

Development– Evaluation committee demands

that she gets half of the fundingfor the new lab from prestigiousgovernment funding agency

– How to start a research lab– How to write a multi-year plan– How to acquire funding from

industry and government fundingagencies

– How to acquire space andequipment

– How to acquire students– How to develop a support network– How to acquire collaborators

within and outside her institution

Her Majesty’s University– Department size: 800 UG / 150 MS / 50

PhD / 75 faculty

Philip (35, PhD)– Reader– Currently not teaching because he

has been included in thedepartment’s RAE (Researchassessment exercise) submission

– Works late and attends manyconferences

– Needs materials that will explainthe basics of computing research tohis Masters and PhD students

– Needs to know who is publishingwhere and who is collaboratingwith who

– Needs to know emerging hot topicsin computing research especially ifthe research council is planning tooffer funding in that area

198

Table 4: A snapshot of the planned content for the Computing Research Methods Multi-Perspective DigitalLibrary (CRM-MPDL). We welcome input on CRM-MPDL content from the larger community.

Researching Computing

– Exemplar materials (papers, funded proposals, reports) using CRM– Materials describing new CRM– Book reviews, CRM survey papers & papers about specific CRM

Integrating Research Across the Computing Curriculum

– Book reviews about teaching CRM– Papers about teaching CRM– Curricular materials, best practice guidelines for teaching CRM– Moderated forums for faculty, students, and staff to share ideas on teaching CRM– A repository of data sets for comparative studies of teaching CRM– Best practice guidelines for undergraduate research in computing– Best practice guidelines for working with students in post-baccalaureate computing research– Best practice guidelines for working with dissertation students in computing– What stats courses should I take? A set of guidelines by computing specialty– Aggregate metrics re: students, faculty, administrators, schools participating in CRM, successful programs– A staging ground for inter-institutional (or intra?-inst.?) research groups

Understanding Research Careers in Computing

– A field guide to academic and industrial research careers in the US and Australasia (+ more to come).– Reviews of books, papers on how to choose and complete a doctoral program– Book reviews, papers on writing for computer science– How to transition from planning a research project into a research program– Designing multi year plans– Setting up a lab– Getting and managing equipment– Recruiting and managing students– Budgeting– Some tough lab issues: visa stuff, etc.– How to get started in academic publishing journals, articles, theses, dissertation, or CVcs– Moderated forums for faculty, students, and staff to explore areas of interest– Where to go for help: faculty, students, schools, industry contacts

Funding Research Across the Computing Curriculum

– Getting and managing “stuff” for your students, program and/or lab.– Funding experiential and exploratory learning programs– Funding research into Computing Research Methods– Best practices for grant writing/proposal/seeking and acquiring/managing funding by funding agency– Funder database– Funding awardee database/list - Who was funded and for how much?

The focus of CRM-MPDL is computing researchmethods, not computing research. Analyzing the con-textualized information needs of our faculty stakeholders,however, reveals a pattern of related needs for both. To fa-cilitate making sense of computing research methods, CRM-MPDL must complement existing efforts to make sense ofthe literature of the various computing research communities(e.g., [11, 33].)

5.3 Enhancing the 2006 framework:Research ethics

As noted earlier, an important finding emerging from the2007 WG‘s work was that the framework for making senseof computing research (see Figure 3) required a further di-mension: “What ethical aspects need to be considered?” In-creasing attention is being paid to the ethics of research andthe rights and responsibilities of those involved. For eachstage of the research we need to ask:

• Who will be involved?

• What rights and responsibilities do they have?

• What ethical clearance is needed before we commencethis stage?

Since the questions need to be asked throughout the re-search life cycle, this dimension is shown super-imposed onthe other four dimensions in the enhanced framework (seeFigure 4).

There is a wide range of participants in Cycles 2 and 3:the WG members, SIGCSE-CRM listserv members and fac-ulty members in the wider community. Each participant canplay more than one role in the research. Each role bringsparticular rights and/or responsibilities:

1. Researchers should: be open and honest about howthey conduct the research; obtain their results withoutany falsification or fabrication; and make clear who issponsoring and/or funding the research.

2. Respondents are those who are interviewed, ob-served, or asked to complete a questionnaire. They

199

I d e n t i f y t h e m e s / p a t t e r n s / q u o t e sC a l c u l a t e n u m b e r sI d e n t i f y t r e n d sE x p r e s s v i a m u l t i m e d i aC r e a t e f r a m e w o r k s / t a x o n o m i e sW h a t d o w e d o w i t h t h e d a t a ?D r a w c o n c l u s i o n sE v a l u a t e r e s u l t sI d e n t i f y l i m i t a t i o n sH a v e w e a c h i e v e d o u r g o a l ?

F i n d o u t w h a t i s h a p p e n i n gD e v e l o p s o m e t h i n g t h a t w o r k sE v a l u a t e a n e x i s t i n g s y s t e m / t e c h n o l o g yC o m p a r e e x i s t i n g s y s t e m sC h a n g e h u m a n b e h a v i o rW h a t d o w e w a n t t o a c h i e v e ?

W h a t e t h i c a l a s p e c t s n e e d t o b e c o n s i d e r e d ?W h o w i l l b e i n v o l v e d ?W h a t r i g h t s a n d r e s p o n s i b i l i t i e s d o t h e y h a v e ?W h a t e t h i c a l c l e a r a n c e i s n e e d e d b e f o r e w e c o m m e n c e t h i s s t a g e ?W h e r e t o c o l l e c t ?F i e l dL a b o r a t o r yC o n c e p t u a lH o w t o c o l l e c t ?R e a dO b s e r v eA s kM e a s u r eE x p e r i m e n tM o d e l

W h e r e d o e s t h e d a t a c o m e f r o m ?

Figure 4: Enhanced framework for teaching CRM.

have the rights: to choose whether or not to partici-pate, to withdraw from the research, to give informedconsent, and to have anonymity and confidentiality.

3. Technical developers should follow the ACM Codeof Conduct and the terms and conditions (CreativeCommons License) of the Greenstone software.

4. Viewers should agree to use the content fairly. Forexample, they should not use materials obtained fromCRM-MPDL without proper attribution.

5. Contributers are those who supply CRM-MPDLwith materials. They have the rights as outlined forrespondents and, in addition, the responsibility to actwith integrity. For example, they must not plagiarizeothers’ work or upload materials for which they havenot obtained the permission of the holder of the in-tellectual property rights. Contributors in the forumsand other interactive community tools should agree toembrace the spirit of civil disagreement and not pub-lish unacceptable content (see [6] for an example ofcommunity guidelines for an online group).

6. CONCLUSIONSThe CRM literature is not currently organized in a coher-

ent fashion. The interdisciplinary nature of our professionencourages a large number of collaborations, but discour-ages the development of a coherent CRM corpus. Our par-ticipatory design community contained both students andfaculty who have not done, or could not see themselves do-ing, research in computing. This pattern presents barriersto integrating CRM into the computing curriculum.

Research skills also vary widely within the computing fac-ulty. In addition to experienced and developing faculty re-searchers, we count among our numbers faculty with little orno expertise in research. Although exceptions exist, intro-ductory classes tend to be taught by faculty with less exper-tise in research, a pattern we suspect may be a factor in earlyattrition of some of the most promising future researchers

to related fields which emphasize early research experiences.To successfully facilitate sense-making among our diversefaculty participants, as well as the rest of the stakeholdersdiscussed in Section 2.1, interaction with CRM-MPDL mustsuit the needs of new and experienced researchers while con-straining neither.

We are using a combination of participatory design tech-niques to contextualize the information needs of comput-ing faculty as related to research methods in computing.Through the personas, listserv, working groups, etc., weare designing detailed perspectives to CRM-MPDL to al-low a multitude of users with widely varying backgroundsto comfortably use the tools created. As an example, con-sider CRM-MPDL’s current content design (see Section 5.2.)The design reflects a broad definition of what is related toCRM, yet broad consensus exists among project participantsconcerning the need for a common resource covering the fulldefinition.

Using personas as proxy users ensures that the design andconstruction of CRM-MPDL is driven by the true needs ofthe users rather than by software contraints or biases ofthe software engineers [14]. Contextualizing the faculty’sinformation needs by using the personas should combat theinevitable tendency towards design drift that is the result ofbuilding tools for one’s own community.

The personas play other roles in the project, as well. Au-diting the personas reinforces participatory design. The au-dit process provides the project principals with an opportu-nity to consult the committed core participatory design com-munity concerning whether a realistic cross-section of usersis captured by the personas. It strengthens the relationshipbetween the project principals and the participatory designcommunity, and attracts new participants. We have foundthat the personas also help with dissemination, helping fac-ulty understand how CRM-MPDL might be useful to themin the future, or to a friend or colleague.

7. CURRENT AND FUTURE WORKFor the near future, the project is furthering integrating

200

research methods into the computing curriculum by pur-suing two goals: implementation and evaluation of CRM-MPDL and continuing calls for participations from the com-puting community.

7.1 Implementation and EvaluationThe project has matured sufficiently that we have begun

CRM-MPDL’s construction and configuration. Greenstone,a well-established open source digital library server with avery large user base, is serving as the base platform. Ms.Okoye is the principal application builder. Dr. Holz andMs. Okoye are collaborating to customize Greenstone to addfunctionality needed for faculty perspectives, such as real-time dialog, collaborative editing, document version control,comments, and so forth. A secondary goal associated withthis Greenstone implementation is to produce professionalcode that can be used in teaching students to read code, asauthentic model teaching code is in great need [7].

We are using layered evaluation to determine how wellCRM-MPDL meets the faculty’s contextualized informationneeds. Once we determine that our personas are typical ofour target users as discussed in Section 3.3, we will exam-ine system usage, specifically, which areas of CRM-MPDLusers are accessing. We will compare our users’ resource us-age patterns to the pattern predicted by the persona mostsimilar to the user. How closely those patterns align willprovide a measure of how well we were able to translate anddesign potential needs correctly based on personas and toincorporate those needs into CRM-MPDL’s design.

At the same time, we will also consider confounding is-sues such as usability. We will carry out usability testingof CRM-MPDL in the spirit of agile software development,resolving confounding issues as they are discovered. Thisapproach will also enable us to analyze any residual effectsof confounding issues such as usability on our results. Ad-ditionally, we will track (in aggregate) the level of partic-ipation of each user over time. In CRM-MPDL, if we aresatisfying our user’s information needs, we expect increasedparticipation from them. One notable sign of this would beincreased interaction with other users that have previouslyaccomplished the goal they are attempting to accomplish.We will provide a free comment area where users are encour-aged to and can leave feedback. With the results gained fromthese analyses, we will validate that CRM-MPDL meets itsusers’ information needs. Ifeyinwa Okoye, one of the au-thors of this report, is a MS student and is doing her thesison this subject. As such, we expect to get an in-depth eval-uation of specifically if, and how, CRM-MPDL meets theusers’ information needs.

7.2 Call for ParticipationAs we move forward with CRM-MPDL, the project will

continue to engage with the computing community to refineand build upon on our understanding of the multiple per-spectives involved in this effort. At this stage of our work,we are seeking participation in three key areas.

Global Design. As discussed in Section 1.3, we are work-ing on a framework based on an extension of the revisedCarnegie Classifications to support a global design for CRM-MPDL. Possible approaches include constructing correspon-dences between classification systems, describing similaritiesand differences, or simply raising awareness of sense-makingprocesses.

Personas. We are seeking feedback on the authenticityof current project personas, as well as suggestions on aspectsof faculty stakeholders that we may have missed. Completeinformation on the current project personas, their institu-tions, information needs, etc., is available on the projectwebsite. As discussed earlier, we are particularly interestedin ways to enrich our design tool such that it is not overlyUS-centric.

Student Perspective Design Cycle. Another effortunder way is the development of the student perspectivesfor CRM-MPDL. A CRM-MPDL WG (possibly for ITiCSE2008) has started to form composed of and directed by stu-dents. Two of the authors, Ifeyinwa Okoye and Kather-ine Deibel, are directly involved in this effort. The studentWG will focus on developing the student (undergraduate,postgraduate, and maybe even K-12) perspective of CRM-MPDL using both personal experiences of its members andby possibly developing a set of student personas and theirrelated informations needs. One goal of the students’ WGwill be generating a broader understanding of students’ in-formation needs from CRM-MPDL.

8. ACKNOWLEDGMENTSFirst and foremost, we would like to thank the SIGCSE

Board for their continued support of this project, most re-cently a SIGCSE Special Projects Grant. We would liketo acknowledge the ongoing support of CSUEB, includinga generous grant from the CSUEB, Foundation and theirongoing hosting of CRM-MPDL. Special thanks to projectmembers Mordecai Ben-Ari and Bruria Haberman for Ronit,and Alan Fekete for volunteering to help adapt the fieldguide for Australasia. Additional thanks goes to the BritishComputer Society and the University of Washington Gradu-ate and Professional Student Senate for helping fund Kather-ine Deibel’s participation in the WG.

9. REFERENCES[1] ACM guide to computing literature.

http://portal.acm.org/portal.cfm, 2007.

[2] B. B. Alexander, J. Foertsch, and S. Daffinrud. Thespend a summer with a scientist program: Anevaluation of program outcomes and the essentialelements for success. Technical report, LEAD Center,University of Wisconsin-Madison, July 1998.

[3] V. L. Almstrum, O. Hazzan, M. Guzdial, andM. Petre. Challenges to computer science educationresearch. In Proceedings of the 36th SIGCSE technicalsymposium on Computer science education, pages191–192, New York, NY, USA, 2005. ACM Press.

[4] V. L. Almstrum and C.-C. Wu. The CSedRes toolbox:A resource to support research in computer scienceeducation. SIGCSE Bull., 25(4):21–26, 1993.

[5] G. Bjerknes, P. Ehn, and M. Kyng. Computers anddemocracy. Aldershot, Avebury, 1987.

[6] Blogher community guidelines. http://blogher.org/what-are-your-community-guidelines.

[7] G. Booch. Readn’, writ’n, ’rithmetic...and code’n.SIGCSE Bulletin, 39(1):197–197, 2007.

[8] J. Brainard. The tough road to better science teaching.The Chronicle of Higher Education, 53(48):A16, 2007.

[9] S. Brint, M. Riddle, L. Turk-Bicakci, and C. S. Levy.From the liberal to the practical arts in american

201

colleges and universities: Organizational analysis andcurricular change. Journal of Higher Education,76(2):151–180, Mar/Apr 2005.

[10] The carnegie classification of institutions of highereducation.http://www.carnegiefoundation.org/classifications/,2006.

[11] L. N. Cassel, R. H. Sloan, G. Davies, H. Topi, andA. McGettrick. The computing ontology project: thecomputing education application. In SIGCSE ’07:Proceedings of the 38th SIGCSE technical symposiumon Computer science education, pages 519–520, NewYork, NY, USA, 2007. ACM Press.

[12] CiteSeer. http://citeseer.ist.psu.edu/, 1997.

[13] A. Cooper. The Inmates Are Running the Asylum :Why High Tech Products Drive Us Crazy and How ToRestore The Sanity. Sams, Indianapolis, IN, USA,1999.

[14] A. Cooper. The Inmates Are Running the Asylum,chapter Designing for Pleasure. Number 9. Sams -Pearson Education, 2004.

[15] P. J. Denning. Great principles in computingcurricula. In SIGCSE ’04: Proceedings of the 35thSIGCSE technical symposium on Computer scienceeducation, pages 336–341, New York, NY, USA, 2004.ACM Press.

[16] B. Dervin. From the mind’s eye of the user: Thesense-making qualitative-quantitative methodology. InB. Dervin, L. Foreman-Wernet, and E. Lauterbach,editors, Sense-Making Methodology reader: Selectedwritings of Brenda Dervin, pages 270–292. HamptonPress, Cresskill, NJ, 2003.

[17] K. Don. Greenstone3: A modular digital library.http://www.greenstone.org/docs/greenstone3/

manual.pdf.

[18] Engineering Information. Compendex.http://www.ei.org/databases/compendex.html.

[19] S. Fincher and M. Petre. Computer Science EducationResearch. RoutledgeFalmer, 2004.

[20] J. Fry and S. Talja. The intellectual and socialorganization of academic fields and the shaping ofdigital resources. J. Inf. Sci., 33(2):115–133, 2007.

[21] R. L. Glass, V. Ramesh, and I. Vessey. An analysis ofresearch in computing disciplines. Communications ofthe ACM, 47(6):89–94, June 2004.

[22] J. Grudin. Obstacles to participatory design in largeproduct development organizations. In D. Schuler andA. Namioka, editors, Participatory Design: Principlesand Practices, chapter 6, pages 99–119. LawrenceErlbaum Associates, Hillsdale, New Jersey, USA, 1993.

[23] J. Grudin and J. Pruitt. Personas, participatorydesign and product development: An infrastructurefor engagement. In Proceedings of the ParticipatoryDesign Conference 2002, pages 144–161, 2002.

[24] Strategic programs for innovations in undergraduatephysics: Project report. Technical report, NationalTask Force on Undergraduate Physics, January 2003.

[25] H. J. Holz, A. Applin, B. Haberman, D. Joyce,H. Purchase, and C. Reed. Research methods incomputing: what are they, and how should we teachthem? In ITiCSE-WGR ’06: Working group reports

on ITiCSE on Innovation and technology in computerscience education, pages 96–114, New York, NY, USA,2006. ACM Press.

[26] A.-B. Hunter, S. L. Laursen, and E. Seymour.Becoming a scientist: The role of undergraduateresearch in students’ cognitive, personal, andprofessional development. Science Education,91:36–74, 2007.

[27] C. C. Kuhlthau. Inside the search process:Information seeking from the user’s perspective.Journal of the American Society for InformationScience, 42:361–371, 1991.

[28] J. Lave and E. Wenger. Situated learning: Legitimateperipheral participation. Cambridge University Press.,1991.

[29] C. Liu and R. Houdek. Teaching computer sciencegraduate students scholarly literature reviewtechniques. In Frontiers in Education Conference, 36thAnnual, pages 14–19, 2006.

[30] A. C. McCormick and C.-M. Zhao. Rethinking andreframing the carnegie classification. Change, pages51–57, September/October 2005.

[31] B. McMurtrie. Europe’s education chief seekstrans-atlantic cooperation. The Chronicle of HigherEducation, 52(25):A39, February 2006.

[32] B. J. Oates. Researching Information Systems andComputing. Sage Publications Ltd., 2006.

[33] A. Pears, S. Seidman, C. Eney, P. Kinnunen, andL. Malmi. Constructing a core literature forcomputing education research. SIGCSE Bull.,37(4):152–161, 2005.

[34] J. Pruitt and J. Grudin. Personas: Practice andtheory. In Proceedings of DUX 2003, 2003.

[35] S. H. Russell. Evaluation of nsf support forundergraduate research opportunities: Draft synthesisreport executive summary. Technical report, SRIInternational, July 2006.

[36] D. Schuler and A. Namioka. Participatory Design:Principles and Practices. Lawrence ErlbaumAssociates, Hillsdale, NJ, 1993.

[37] SIGCSE-CSRM. SIGCSE Committee on TeachingComputer Science Research Methods.http://acc.csueastbay.edu/∼csrm/.

[38] D. I. K. Sjøberg, T. Dyba, and M. Jørgensen. Thefuture of empirical methods in software engineeringresearch. In FOSE ’07: 2007 Future of SoftwareEngineering, pages 358–378, Washington, DC, USA,2007. IEEE Computer Society.

[39] T. Sumner and M. Marlino. Digital libraries andeducational practice: a case for new models. In JCDL’04: Proceedings of the 4th ACM/IEEE-CS jointconference on Digital libraries, pages 170–178, NewYork, NY, USA, 2004. ACM Press.

[40] D. P. Truex, R. Baskerville, and H. Klein. Growingsystems in emergent organizations. Commun. ACM,42(8):117–123, 1999.

[41] R. L. Van Horn. Empirical studies of managementinformation systems. Data Base, 5(2-3-4):172–182,1973.

[42] R. Whitley. The Intellectual and Social Organizationof the Sciences. Clarendon Press, Oxford, 2nd edition,

202

2000.

APPENDIX

A. TABLE OF ACRONYMS

A&S Arts & SciencesACM Association for Computing MachineryBOF Birds of a FeatherCC Community CollegeCE Computer EngineeringCOP Community of Practice / Communities of

PracticeCRM Computing Research MethodsCRM-MPDL

Computing Research Methods Multi-Perspective Digital Library

CS Computer ScienceCSEd Computer Science EducationCSRM Computer Science Research MethodsCSUEB California State University, East BayD.Eng. Doctor of EngineeringDL Digital LibraryDSc Doctor of ScienceESU Enormous State UniversityGRV Glass, Ramesh and VesseyICC Ivy Covered CollegeISc Information SciencesIS Information SystemsIT Information TechnologyITiCSE Innovation and Technology in Computer

Science EducationLPP Legitimate Peripheral ParticipationMBA Master of Business AdministrationMCIS Master of Computer Information ScienceMS Master of ScienceNSF National Science FoundationOAI-PMH Open Archives Initiative Protocol for

Metadata HarvestingREU Research Experiences for UndergraduatesR’RU Research ’R UsSE Software EngineeringSIG Special Interest GroupSTEM Science, Technology, Engineering and

MathematicsUG UndergraduateUR Undergraduate ResearchWG Working Group

203