Eliciting ethical and social values in health technology assessment: A participatory approach

lable at ScienceDirect

Social Science & Medicine xxx (2011) 1e10

Contents lists avai

Social Science & Medicine

journal homepage: www.elsevier .com/locate/socscimed

Eliciting ethical and social values in health technology assessment:A participatory approach

Yvonne Bombard a,*, Julia Abelson b, Dorina Simeonov b, Francois-Pierre Gauvin c

aDepartment of Health Policy Management and Evaluation, University of Toronto, 155 College Street, 4th Floor, Toronto, Ontario, Canada M5T 3M6bCentre for Health Economics and Policy Analysis, Department of Clinical Epidemiology and Biostatistics, McMaster University, CanadacNational Collaborating Centre for Healthy Public Policy, Institut national de santé publique du Québec, Canada

a r t i c l e i n f o

Article history:Available online xxx

Keywords:Health technology assessmentEthicsValuesPublic engagementPublic involvementDeliberative methodsHealth policyDecision-makingSocial values

* Corresponding author. Tel.: þ1 416 978 7864; faxE-mail addresses: [email protected]

mcmaster.ca (J. Abelson), [email protected] (F.-P. Gauvin).

0277-9536/$ e see front matter � 2011 Elsevier Ltd.doi:10.1016/j.socscimed.2011.04.017

Please cite this article in press as: Bombarapproach, Social Science & Medicine (2011),

a b s t r a c t

Despite a growing consensus that ethical and social values should be addressed in health technologyassessment (HTA) processes, there exist a variety of methods for doing so. There is growing interest ininvolving citizens in policy development to ensure that decisions are legitimate, and reflect the broadsocial values of the public. We sought to bring these issues together by employing a participatoryapproach to elicit ethical and social values in HTA. Our primary objective was to elicit a set of ethical andsocial values from citizens that could be used to guide Ontario’s HTA evidentiary review and appraisalprocess. A secondary objective was to explore the feasibility of using participatory approaches to elicitthese values.

A 14-person Citizens’ Reference Panel on Health Technologies was established to provide input to theOntario Health Technology Advisory Committee in developing its recommendations. A mixed methodsapproach was used where informed, deliberative discussions were combined with pre- and post-ques-tionnaires, which assessed the relative importance of various ethical and social values as well as theirstability over time.

Over the course of five meetings, panel members progressed toward the identification of a set of corevalues e universal access, choice and quality care. These values were consistently prioritized as the corevalues that should be considered in the evaluation of health technologies and ensuing recommendations.Sustained and deliberative methods, like a citizens’ panel, offer a promising approach for eliciting ethicaland social values into HTA.

� 2011 Elsevier Ltd. All rights reserved.

Introduction

Health technology assessment (HTA) involves the systematicevaluation of evidence for a particular health technology to informhealth policy decision-making. HTA first emerged in the 1970s tocounter the ‘uncontrolled introduction of new technologies intohealth care practice’ (ten Have, 2004) (pg. 72), and to ensure thathealth technologies were developed, diffused and used in a safeand cost-effective manner (Banta, 2003). In its original conception,HTA was meant to “examine the short- and long-term socialconsequences of the application or use of technology” (Office ofTechnology Assessment, 1976). However, it is only recently thatethical and social values have been considered by HTA agencies;

: þ1 416 978 7350.(Y. Bombard), abelsonj@

m (D. Simeonov), fpgauvin@

All rights reserved.

d, Y., et al., Eliciting ethicaldoi:10.1016/j.socscimed.2011.

and even still, only a minority (17%) of HTA reports address ethicalissues (DeJean, Giacomini, Schwartz, & Miller, 2009; Reuzel, 2004).

Addressing ethical and social values in HTA presents numerouschallenges that stem fromthe plethora ofmethods that exist and thelimited ethics capacity within HTA agencies (Autti-Rämö & Mäkelä,2007; Lehoux &Williams-Jones, 2007). Indeed, the very selection ofa method for ethical analysis is a value-laden decision (Saarni,Hofmann, Lampe, Luhmann, Mäkelä, Velasco-Garrido et al., 2008).Traditionally, HTA analysts employ three broad methodologicalapproaches to address ethical and social values in HTA: conductingprimary research, performing secondary analysis of publishedliterature on ethical and social values; and, using expert advice(Lehoux&Williams-Jones, 2007). Further, there is a rangeof analyticapproaches available to ethics experts including: casuistry (i.e.,where moral challenges are compared to similar solved cases);coherence analysis (i.e., where a set of arguments are examined forinternal consistency), principalism (i.e., where the basic set ofbioethical principles are applied to moral challenges), wide reflec-tive equilibrium (i.e., where multiple reflective judgments are used

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e102

to reach an agreement), and interactive participatory approaches(i.e., the use of stakeholders to reach an agreement) (EuropeanNetwork for Health Technology Assessment (EUnetHTA), 2007;Hofmann, 2005; International Network of Agencies for HealthTechnology Assessment (INAHTA) Ethics Working Group, 2005;National Institute for Health and Clinical Excellence, 2008; Saarniet al., 2008). Despite a growing consensus among scholars andHTAproducers that consideration of ethical and social values shouldbecome standard in HTA (Velasco, Perleth, Drummond, Gurtner,Jorgensen, Jovell et al., 2002), there exist a variety of methods fordoing so. In fact, there have been increasing calls to experimentwithmethods to integrate these considerations in HTA (Lehoux &Williams-Jones, 2007).

Equally, there is a growing interest in involving a broader rangeof stakeholders in public policy development and HTA-relatedpolicy more specifically e including patients and service users (or‘consumers’) as well as the broader citizenry (Abelson, Giacomini,Lehoux, & Gauvin, 2007; Bridges & Jones, 2007). This enthusiasmstems from a need to ensure that decisions are informed, trans-parent and legitimate (Gauvin, Abelson, Giacomini, Eyles, & Lavis,2010; Pivik, Rode, & Ward, 2004), but also in recognition of theongoing challenge that incorporating value judgments into the HTAprocess presents for HTA producers (Gauvin et al., 2010). Indeed,the balance of benefit and harms is inherent to the HTA process andrelies on making value judgments on which harms and benefitsto evaluate (ten Have, 2004). One approach to remedy this is toincorporate public input in HTA reports, which would serve toenhance both the relevance and acceptability of ensuing decisions(Calnan, 1998; Coulter, 2004; Pivik et al., 2004; Sorrenson,Drummond, Kristensen, & Busse, 2008). In fact, the National Insti-tute for Health and Clinical Excellence’s (NICE) has developed a setof social value judgments to guide its decision-making, based inpart on proceedings of the Citizens’ Council (National Institute forHealth and Clinical Excellence, 2008).

We aligned these complementary but distinct areas of activity byexperimenting with a particular method e an interactive partici-patory approach drawn from the field of public engagement e toelicit ethical and social values to inform the HTA policy process inOntario, Canada. Our primary objective was to elicit a set of ethicaland social values from citizens that could be used to guide Ontario’sHTAevidentiary reviewandappraisal process. A secondaryobjectivewas to explore the feasibility of using participatory approaches toelicit these values. In particular, we explored the use of structureddeliberation to elicit a set of stable and consistent ‘core’ values overtime (Hutcheon, 1972; Rokeach, 1973, 1979).

Fig. 1. OHTAC-MAS Evidence-based Review Process. Figure reprinted with permission from F(Copyright � 2009 Cambridge University Press).

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Methods

Study context

The Medical Advisory Secretariat (MAS) and the Ontario HealthTechnology Advisory Committee (OHTAC) have had a long-standinginterest in stakeholder engagement and incorporating ethical andsocial values into their HTA and policy process. As set out in theirframework for evidence-based recommendations in Ontario,OHTAC and MAS endeavor to incorporate ethical and social valuesas part of their criteria or ‘decision determinants’ (Johnson, Sikich,Evans, Evans, Giacomini, Glendining et al., 2009). The process fordoing so, however, is in the early stages of development. Between2006 and 2010, several OHTAC subcommittees have been chargedwith the task of making recommendations regarding the ways inwhich OHTAC could: (1) engage with stakeholders, and (2) inte-grate ethical and social values in their HTA and recommendations.Ethical values were conceptualized as the “potential ethical issuesinherent in using or not using the technology”, while social valueswere characterized as “broadly shared values in society which bearon the appropriate use and impact of the technology” (Johnsonet al., 2009). Indeed, the distinction between these values maynot be straightforward as they blend traditionally distinct disci-plines of bioethics with social sciences and science technologystudies (Lehoux & Williams-Jones, 2007). However, these defini-tions are aligned with existing academic conceptualizations, whichdistinguish ethical issues as principled, value-laden, normativeassessments of HTA and the use of technologies, from social issuesthat make the potential unintended consequences of their usesexplicit (Giacomini, Hurley, Gold, Smith, & Abelson, 2004; Heitman,1998, 1999; Lehoux & Williams-Jones, 2007; van der Wilt, 2004).

In its final report, the public engagement subcommittee ratifiedthe need for public consultation, identified stakeholder categories,and proposed a framework to incorporate public input (Fig. 1)(Public Engagement Subcommittee of the Ontario HealthTechnology Advisory Committee, 2007). Specifically, the commi-ttee identified 3 stages to incorporate public input: i) at the vignettestagewhere the public would have the opportunity to provide inputon the research questions guiding the HTA; ii) at the draft recom-mendation stage when the public could help shape the recommen-dations resulting from the HTA; and the public comment stagewhenthe recommendation is in its final stages of review. A secondsubcommittee, charged with the task of identifying methods forintegrating ethical and social values into the HTA process, has onlyrecently begun its work. Our project emerges at the nexus of these

ig. 1, Johnson et al., 2009 International Journal of Technology Assessment in Health Care

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e10 3

lines of inquiry and investigates the questions of: how to elicitethical and social values inHTA andhowto engagewith the “generalpublic/citizens/taxpayer”, a categoryof the stakeholder engagementplan,meant to reflect theperspectives of the “friends or neighbors ofpatients, employers, members of local or cultural communities, andcitizens” (Public Engagement Subcommittee of the Ontario HealthTechnology Advisory Committee, 2007).

Participant recruitment

A 14-person Citizens’ Reference Panel on Health Technologieswas established from 2009e2010 as part of an independent studyto provide input to OHTAC at various stages in the HTA process(Fig. 1). The study was approved by the Hamilton Health Sciences/McMaster University Faculty of Health Sciences Research EthicsBoard.

Demographic and geographic balancewas sought in creating the14-person Panel. We used a stratified, random sampling approach,where 3500 randomly selected Ontario residents were mailed aninvitation letter, information sheet and postage-paid responseform. From the 3500-person sample, 163 responses of expressedinterest in panel membership were received. Based on a ‘civiclottery system’ (Dowlen, 2008), 14 panel members were blindlyselected from the respondent pool, sorted by region or ‘Local HealthIntegration Network (LHIN)’, and stratified by gender and agebracket. The Panel consists of 5 women and 9 men, one memberfrom each LHIN, and includes: two members between ages 18e24,four between 25e39, four between 40e54, two between 55e70,and one aged over 71 (Table 1). One woman (aged 40e54) and oneman (aged 25e30) withdrew from the Panel following the firstmeeting.

Table 1Demographic characteristics of panel members.

Total (n ¼ 14)

n %

GenderFemale 5 36%Male 9 64%

Age18e25 1 7%25e39 4 29%40e54 5 36%55e70 3 21%71þ 1 7%

Marital Status (n ¼ 8)Single/separated/divorced/widowed 2 25%Married/common law/living with partner 6 75%

Education (n ¼ 13)Highschool & below 2 15%Some college or university & above 11 85%

Employment Status (n ¼ 13)Employed 9 69%Unemployed/Retired 3 23%Student 1 8%

Income (n ¼ 12)<$20,000 2 17%$20,000 e $40,000 2 17%$40,000 e $60,000 5 42%$60,000 e $80,000 3 25%>$80,000 0 0%

Ethnic Background (n ¼ 13)Canadian 7 54%European 3 23%European Canadian 1 8%South Asian 1 8%Asian 1 8%

Missing values are excluded.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Deliberative discussions

Panel members were engaged in informed, facilitated discussionand values elicitation through five 1-day structured deliberationsessions on two major topics. The first, sustained topic was theelicitation of ethical and social values into the evaluation of healthtechnologies (i.e., what are the core values that should guide MAS-OHTAC evaluations of health technologies, when and by whomshould this be done?). The second was a series of technology-specific discussions, where the Panel reviewed selected healthtechnologies at various stages in the MAS-OHTAC review process(i.e., vignette stage, draft recommendation stage and publiccomment stage) to provide input to OHTAC deliberations. Thefollowing five technologies were being concurrently reviewed byMAS-OHTAC, and were selected by MAS-OHTAC for public input:colorectal cancer screening, percutaneous aortic value replace-ment, breast cancer screening, gene expression profile testing, andserologic testing for celiac disease. The former question is the focusof the paper, which was more fully explored through technology-specific discussions.

Material on each deliberation topic was circulated one week inadvance of each meeting using a workbook format which includedtopic summaries geared toward a lay audience and a set ofdiscussion questions. Relevant newspaper and/or summary articleson the topic were also pre-circulated. The agenda for each day-longmeeting beganwith a discussion on each topic and a brief summarypresentation by the meeting facilitator (JA) or guest presenter,followed by a Q&A session and either externally-facilitated or self-facilitated discussion around pre-circulated questions in large andsmall groups.

Data collection and analysis

A mixed methods approach was used where informed, delib-erative discussions were combined with pre- and post-question-naires. This study was driven by an inductive process, where valuesthat emerged from deliberations were complemented by findingsfrom the questionnaire (Morse, 1991).

Discussions were transcribed verbatim, and transcripts weremanaged and analyzed using QSR NVivo 8. A codebook was devel-oped by YB and DK, which encompassed the moral questions, thetechnologies that were presented to the panel and general reflec-tions and tensions that arose in the context of the deliberativediscussions. Examples of these codes include: “patient autonomy”,“challenges to social values and arrangements”, “breast cancerscreening”, “reaction to Hofmann’s questions”, “conflicting values”,respectively. The codebook was later refined based on emerginganalysis. Discussions were jointly analyzed using content analysis,based on the principles of constant comparison (Strauss, 1998) andqualitative description (Sandelowski, 2000, 2010), where propertiesand dimensions of the codes were described, and then contrastedwith emerging data and examples. The results of these qualitativeand quantitative analyses were presented to the Panel at the lastmeeting as a member-checking exercise, and to encourage furtherdiscussion and elaboration of the core values. There was generalagreement with the analysis; and, further insights and discussionshave been incorporated in the following results. We did not seek toachieve group consensus using conventional consensus-seekingmethods. Indeed, values-based reasoning that seeks commonground and agreement rather than formal consensus is the goal ofmost deliberative exercises (Karpowitz & Mansbridge, 2005).

Questionnaires were developed by the research team to assessthe relative importance, assigned by panel members, to differentethical and social values to the HTA process (objective 1) as well asthe stability of their views over time (objective 2). Specifically,

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e104

panel members were asked to consider a list of moral questions andto identify their ‘top 3 picks’ among the list, as an alternative toa ranking exercise. Appendix A provides a sample questionnaireexcerpt. In addition, panel members were asked to rate theimportance of the top ethical and social values identified from theprevious meetings. The ratings from this question were used toexamine the relative stability of Panel members’ views over time.Appendix B provides a sample questionnaire excerpt. Panelmembers were given the opportunity to add or amend values,which were then incorporated into the questionnaire andcomprised its validation. Pre- and post-meeting questionnaireswere analyzed with descriptive statistics as well as repeatedmeasures tests (e.g. Wilcoxon signed ranks test), using SPSS(version 17).

Results

Panel members deliberated on five health technologies atvarious stages of the MAS-OHTAC process. In addition to providinginput on these selected health technologies, panel members wereasked to reflect on the ethical and social issues associated with theuse and diffusion of these health technologies. Over the course offive meetings convened over the two-year study, panel membersprogressed toward the identification of a set of core ethical andsocial values e universal access, choice and quality care e at stake inevaluating health technologies and in forming recommendationsfor their use in the health system.

In search of values

At first, panel members struggled in reflecting on the ethical andsocial issues inherent in the health technologies presented to them.They admitted to feeling ill equipped with the task and requestedmore guidance for their deliberations:

“I’mwondering, if it would be.to be able to look at a templatethat we might use for all of the proposals that come throughwith some standardized questions around societal impact.a broader framework at each stage of the process. I think thatwould help me a lot” (Paula*, age range: 40e54) (*Aliases havebeen used to protect participants’ anonymity.)

In response to this request, we drew on Hofmann’s list of 33‘moral’ questions for HTA (Hofmann, 2005) because it is widelyused as a guide to address ethical issues in HTA, as supported byseveral international HTA working groups and experts (EuropeanNetwork for Health Technology Assessment (EUnetHTA), 2007;International Network of Agencies for Health TechnologyAssessment (INAHTA) Ethics Working Group, 2005; Saarni et al.,2008). Panel members were presented a shorter list of 20verbatim questions (exclusions were made to keep the list toa manageable size and to address lay comprehension concerns).They were categorized as placing either ethical or social values atstake, and were sorted to align with OHTAC’s definitions of ethicaland social values (Johnson et al., 2009). For example, ‘ethicalquestions’ reflected “potential ethical issues inherent in using ornot using the technology” (e.g., questions related to the moralconsequences of using the technology in preserving autonomy,integrity, dignity, etc.), while ‘societal questions’ reflected “broadlyshared values in society which bear on the appropriate use andimpact of the technology” (e.g., questions that pertained to thedistribution of resources, to commercial interests, to religious andcultural values, etc.). Panel members were then asked to reflect onthe questions’ utility and applicability in evaluating health tech-nologies and had the opportunity to add or remove values to the listof those up for consideration.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Developing core values

An iterative process ensued among Panel members, where theyused Hofmann’s list to elicit their own fundamental values and thenmapped these principles onto their evaluation of specific technol-ogies. One panel member discussed the importance of developinga set of core values specific to the panel:

“Well, based onwhat I’mhearing fromeveryone. that is the endgoal, which I think is to get the information and the technologyout to everyone, then we actually should, that’s a very broaddefinition, narrow it down towhatwe think it is and then executefrom there because how else are we going to attack a populationthis bigwithout understandingwhat our values are andwhat ourcore ethical values are as well.” (Ken, age range: 18e24).

Through further deliberation, core values began to emerge amongpanelmembers.Theyreflectedonwhetherandhowthe ‘list’addressedthe core values of Canadian health care, and in doing so, developeda principled stance as to what constituted their own core values:

“Thank you for this, I think it’s brilliant, the outline for thesocietal and ethical values, just because I’m just hearing some ofthe things that you’re saying as far as the option of consent, thechoice piece, those are the three fundamental, core values of ourhealth care system. It is: universal access, quality care, andchoice of options.” (Paula, age range: 40e54).

Using Hofmann’s questions, members distilled what theyperceived to be the fundamental values at stake for the health caresystem in adopting new health technologies. Indeed, the corevalues of: universal access, choice and quality care developed intorecurring themes throughout their discussions, two of whichmapped onto their pre- and post-questionnaires responses(universal access and choice).

Citizens’ core values in guiding HTA

An interesting and consistent pattern emerged where the corevalues of ‘universal access’ and ‘choice’ became dominant themesthroughout the deliberations, across technologies, as well asexplicit values frequently endorsed in their questionnaires. That is,the ‘distribution of health care’ and ‘patient autonomy’ were themost consistently supported ‘top picks’ by the majority of panelmembers across and within meetings, and mapped onto thethemes that dominated the panel’s deliberations: ‘universal access’and ‘choice of options’, respectively. The third theme that domi-nated deliberations and emerged as an important value for thepanel members was ‘quality care’. However, direct triangulation ofthese data was not possible because ‘quality care’ was not includedas part of Hoffman’s original list of ‘moral questions’. Nonetheless,some overlap was apparent in the way members linked ‘qualitycare’ with physicians and their frequent support of ‘physician-patient relationships’ as ‘top picks’ in the questionnaires.

Qualitative resultsUniversal access. The issue of equal access to, and distribution of,health care was a dominant topic discussed by the panel. Amongthemany issues, members were primarily concerned about the lackof access to family physicians as well as the increasing disparity inaccess to health care.

The physician shortage in Ontario was one of the main concerns,especially in rural communities. Panel members referred to individ-uals that live inrural areasas “orphanpatients”and suggested that theshortage of physiciansmay preclude some fromaccessing a physicianand may even place limitations on individuals’ choice of residence.

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e10 5

Aside from geographic location, members also felt that incomeand level of education determined one’s knowledge of, and thusaccess to, available technologies:

“To me here in Ontario, we have the haves and have-nots. Wehave the educated populationwho might be listening to that TVshowor reading it in a magazine or whatever and running downto the drug store and getting that and being proactive. But wehave a lot of people that are just going to miss it entirely whoalso need the service.” (Nancy, age range: 55e70).

Members felt that one’s occupation, family composition andgeneral demographic profile favored particular citizens’ access toscreening technologies over others. One member felt that thisdisparity challenged the principle of universality:

“Well universality is a nice idea and education is one of thosethings. I would like to see the demographics of the people beingscreened. What’s their education level? Where are they living?What kind of job do they have? . Who are these people beingscreened, that are volunteering for it? At an age earlier than 50,a 30 year old walks into the doctor’s office and says I wantamammogram, her parents are probably highly educated and sois she. I’m not saying probably there’s a good chance of itthough.” (Larry, age range: 40e54).

Panel members held firm positions on the value of equal distri-bution of health care as well as the need to consider this principle inthe evaluation of new technologies. They maintained that thegovernment had a responsibility to distribute technologies equally:

“I guess I think that the Ontario government has to ensure thatthat technology is going to be utilized by everybody, not just themiddle class who may be better off, have a doctor or have thewisdom or education to read about, look into it. That’sthe obligation.” (Nancy, age range: 55e70).

As a matter of principle, members affirmed the importance ofconsidering the distribution of health care when evaluatingtechnologies:

“It’s aboutusasking thesequestions, if eachtimeanewtechnologycomes up, ifwe continue to ask from this framework, I thinkwe’recovering so much ground as far as not missing any representedcultural religious gendered community, nor as far as the choice,the options based on statistical data.” (Paula, age range: 40e54).

The primacy of ensuring universal access to health care and newtechnology was echoed as a core value across all meetings. Theshortage of physicians and socio-economic disparities among thepopulation were acknowledged as the most challenging obstaclesto ensure health equity.

Choice of options. In articulating their values, panel members crit-ically assessed the meaning of choice and autonomy. They definedsix types of information required to enable informed choice: thebenefits, the physical and psychological risks, information aboutthe procedure, time commitment and turn around time for results,comparison of approaches taken in other jurisdictions, and infor-mation about the effectiveness and incidence rates. Overall, citizensvalued a balanced informed consent process that is transparentabout risks and benefits, as well as the available options, in order tomobilize informed choices.

However, panel members acknowledged that exercising one’schoice might create challenges. This became clear in their discus-sions about cancer screening, where participation, or use ofa particular technology, could be perceived as mandatory. In oneinstance, panel members reflected on the breast screening

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

programs and wondered whether its routinization threatenspatient autonomy:

“When you go into one of these clinics or you go to see your careprovider and they set you up for one of these examinations thatyou’re on a deadline, you are basically now booked into anappointment and as that, as such, you feel committed to attend.Now the choice has been removed in most cases because thewoman is going to attend this whether shewants to or not. She’snot even going to consider it anymore; it’s like going toa doctor’s appointment.” (Bob, age range: 40e54).

Members worried about jeopardizing relationships withproviders when exercising choice or seeking alternatives, especiallywhen a particular modality for screening was the only recom-mended option:

“That choice piece is, of course, it’s a bit of a fine line or a double-edged sword, so to speak. And the implications - if it’spopulation-based e is there a choice [between modalities]? Dowe really have a choice to take that test? And if wedo,who dowerun into conflict with? Do we run into conflict with our doctor?Our pharmacist? The whole health network? So do we reallyhave choices? So that’s the ethical question, is: arewe inpotentialconflict with that very person, or people or system, that we’redependent upon for our health?” (Paula, age range: 40e54).

Further, the balancing of risks and benefits in assessing tech-nologies was considered both a large scale and personal process, inwhich members consistently prioritized choice:

“I think it comes down to asking ourselves do the benefits ofa given screening program outweigh the risks of that givenscreening program? And yeah when that day comes it will comedown to the individual woman’s choice absolutely.” (Frank, agerange: 25e39).

Ultimately, members reasoned that having the freedom tochoose e whether to participate in a program or among technol-ogies e respected individual values and religious or cultural beliefs.

Quality care. ‘Quality care’ emerged as the third core value in panelmembers’ discussions. It was routinely discussed in the context ofpatient-physician relationships because physicians were viewed asthe primary access point to technologies.

Panel members portrayed ‘quality care’ as the product oftrusting patient-physician relationships established, or facilitated,by the availability of high quality information, and choices availablewithin collaborative decision-making models. As one memberdeclared: “I don’t know how you collaborate with someone youdon’t trust. And trust with your physician I think is the biggest thingyou’ve got.” (Benjamin, age range: 55e70) Another memberexplained how physicians facilitate ‘quality care’ by being open tobeing questioned, have the patient seek another opinion andcollaborate with the patient on decisions:

“They [physicians] are going to be challenged. People are notgoing to want to do what they are being told to do by thatindividual for whatever reason, and that individual either needsto have an alternative to say, ‘You know what, here’s somebodyyou can go and speak to’, or, ‘You know what, valid question,these are the reasons why you really should do it.’” (Benjamin,age range: 55e70).

Conversely, a sense of distrust emerged that could negativelyaffect quality care. Quality care could be threatened when physiciancredibility was challenged or when the quality of the informationphysicians provided to patients was questioned:

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e106

“How does the person who is on the patient’s seat, determinethat this information that is coming from the service provider orhealth care provider is qualified enough to make that decision?”(John, age range: 25e39).

Another potential source threatening quality care was theinternet. Members acknowledged the increasing role the internetplays in equipping patients with information and the sense ofempowerment over their health care and decision-making.However, they also admitted that this dynamic could degrade thesense of trust in physicians, and the quality of their care:

“As the public becomes more and more educated on thesequestions, the internet becomes God’s book and anything youfind on the internet must be true. They’ve lost a lot of the faiththey had in the medical profession and its accoutrements to dothe best they can for them. we’re sort of stepping over ourmedical doctor to a degree and demanding things that we reallydon’t know about.” (Bernard, age range: 71þ).

Access to care was considered another potential barrier chal-lenging quality care. Members worried that the physician shortagemay limit patients’ sense of freedom to seek a second opinion ortheir ability to decline a doctor’s recommendation. One memberdescribed how the limited access to physicians may impact qualitycare:

“Really tough the way we have it nowadays you fall in to thehands of the medical profession[al], your choices are gone. Theywill tell you what you’re going to do. And if you object to it, theytell you find [another doctor].” (Benjamin, age range: 55e70).

In these ways, the core values of equal access and choice wereinterwoven with trust to characterize ‘quality care’.

Quantitative resultsPanel members were asked to consider the list of moral ques-

tions adapted from Hofmann (2005) (Fig. 2) and to identify their‘top picks’ over the four meetings, following the introduction of thelist. Across the pre- and post-questionnaires, 70e90% of panelmembers selected the question relating to the distribution ofhealth care, 50e80% selected the question relating to patientautonomy, 60% preferred the question relating to the representa-tiveness of users, 50e60% chose questions relating to humanrights, human dignity or human integrity, while 40e60% rated thequestion about patient-physician relationship, as their ‘top picks’.Their top picks fluctuated slightly between technologies; however,the questions that pertained to ‘the distribution of health care’ and‘patient autonomy’ were the questions most consistently sup-ported by the majority of panel members across and withinmeetings (Fig. 2).

In fact, the priority given to these values remained consistentover time. We found that there were no significant differences intheir rankings before and after each meeting, as shown below in anassessment of pre- and post-meeting surveys for meetings 4 and 5(We chose these two meetings because these were the question-naires in which we repeated the same question pre- and post-meeting.) (Tables 2 and 3).

In sum, the values of universal access, choice and quality carewere highly prioritized through the course of the study, suggestingthat this may be a group of core values that citizens feel areimportant and should be considered in the evaluation of healthtechnologies and ensuing recommendations. While these valueswere generally encompassing of the ethical and social values atstake across all technologies, panel members noted the need toidentify relevant moral issues on a case-by-case basis, as somevalues may be more broadly applicable than others.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Discussion

Despite slight variances across technologies, ‘universal access’,‘choice’, and ‘quality care’were identified as core values that shouldbe considered in the evaluation of health technologies and ensuingrecommendations. More specifically, these core values suggest thatdecisions regarding the use and diffusion of health technologiesshould be guided by the principles of equitable access to technol-ogies and the provision of choice to individuals, facilitated throughtrust-based relationships between patients and providers. Publicengagement offers an informed and participatory approach toeliciting ethical and social values in HTA. Deliberation about the useand diffusion of new health technology fostered a process ofmaking public values explicit. This participatory process allowedmembers to find common ground around trade-offs and collec-tively articulate values to guide decision-making. Our resultsdemonstrate that participatory approaches, which employ sus-tained deliberations, are a feasible method to elicit a core set ofethical and social values inherent in health technology assessment.

These findings are consistent with similar work in which ordi-nary citizens were charged with the task of articulating basic valuesand debating these by formulating cogent arguments andacknowledging competing moral considerations (Chafe, Merali,Laupacis, Levinson, & Martin, 2010; De Vries et al., 2010; Scully,Banks, & Shakespeare, 2006). The articulation of values throughpublic engagement processes is also supported by the ‘SocialValue Judgements’ guidance developed by NICE. While based inpart on reports from their Citizens’ Council, NICE’s guidanceemphasizes key principles e human rights, discrimination andequality e which ought to be adhered to in the development ofNICE guidance (National Institute for Health and ClinicalExcellence, 2008). Moreover, the particular values advanced byour Citizens’ Panel align with values discourse portrayed in Cana-dian health care reform documents, public engagement studies andpublic health ethics frameworks. Among Canadian health carereform proposals, policymakers evoke a range of ‘values’ that rangefromwell being, quality of life, quality of care and services to equity,individual rights, dignity and choices (Giacomini et al., 2004).Similarly, another public engagement study has found that trustemerged as a Canadian health care value, which was grounded inboth the health system as well as in individual health careproviders (Abelson, Miller, & Giacomini, 2009). Further, publichealth ethics frameworks advance the principles of: collectivewelfare, individual autonomy, distributive and procedural justice,health disparities and public engagement in decision-making(Childress, Faden, Gaare, Gostin, Kahn, Bonnie et al., 2002;Daniels, 2000; Kass, 2001; Upshur, 2002). Thus, existing frame-works and participatory approaches support the particular valuesadvanced by our Citizens’ Panel and the use of deliberative publicengagement processes to elicit core values inherent in healthtechnology assessment decision-making.

While our results shed light on a set of ethical and social valuesthat may be used to guide technology assessment, they also raisefundamental questions regarding: (1) the role of public input inethics and in HTA; and (2) how public input should be incorporatedinto the HTA and appraisal process.

First, controversy exists regarding who has the authority to offerethical input (Benatar, 2006). Whether the ‘public’ or ordinarycitizens possess the authority or skill to engage as moral analysts isa matter of debate (Mullen, 2008). Likewise, some scholars ques-tion whether public input represents legitimate evidence for HTAor whether they have sufficient expertise or experience with thehealth condition or technology being assessed (Cleemput, Leys,Ramaekers, & Bonneux, 2006; Goven, 2008). Further, commenta-tors question whether ethical analysis provided by members of the

and social values in health technology assessment: A participatory04.017

Fig. 2. Pre- & post- meeting ‘Top Picks’ of moral questions adapted from Hofmann (2005).

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e10 7

public is sufficiently inclusive, and whether the perspectives ofmarginalized or minority groups are included (Burgess, 2004).Thus, challenges arise as to how much ethical weight should begiven to public values.

The corollary is how to arrive at an ethical policy decision.Ultimately, this question falls in the chasm of how to balancedescriptive ethics (i.e. what people believe) with normative ethics(i.e. what ethical principles oblige), which has traditionally beenconsidered irreconcilable (Kenny & Giacomini, 2005). However,using public engagement methods to elicit ethical values may beconsidered a bridge between descriptive and normative ethics.Although our participatory approach is rooted in a ‘descriptiveethics level’, where empirical methods were used to ask whatpeople believe, the attempt and product was an explicit

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

identification of core values and principles, which may be analo-gous to the aims of normative ethics, where descriptive consider-ations are drawn upon to formulate concrete advice (Kenny &Giacomini, 2005). Thus, as is the role of ethics in policy, usingpublic engagement to explore ethics can identify value issues,clarify meaning and broaden perspectives (Kenny & Giacomini,2005), toward the ultimate goal of informing the full complexityof the issues and enhancing the decision made. Just as the HTAdimensions of efficacy, effectiveness or cost-effectiveness cannotindependently determine policy, ethical input from the public caninform, but not establish, the policy decision.

A second issue involves how these values, and public input morebroadly, should be incorporated into the HTA process. Of the fewstudies exploring these issues, they conclude that there is little

and social values in health technology assessment: A participatory04.017

Table 2Attitudes toward the importance of the top ethical and societal values identified in meeting #4.

PRE-MEETING POST-MEETING P-VALUEa

n Mean SD n Mean SD

Top Societal Values Identified:Demonstrating respect for religious, social or cultural convictions 9 3.56 1.74 9 3.11 1.90 0.74Ensuring that technologies are aligned with social values and arrangements 9 2.78 1.64 9 2.00 1.12 0.45Evaluating how the technologies relate to more general challenges of modern medicine 9 2.33 1.00 9 2.22 1.20 0.89Evaluating how the technologies affect distribution of health care 9 1.56 0.73 9 1.56 0.73 1.00Top Ethical Values Identified:Ensuring that individual autonomy is respected 9 1.78 0.83 10 1.90 1.66 0.67Protecting the physician-patient relationship 9 2.11 1.17 10 2.10 0.88 1.00Demonstrating respect for the conception of certain persons 9 2.44 1.42 10 2.70 1.77 0.85Ensuring that human dignity is respected 9 1.33 0.71 10 1.90 0.74 0.06

A 7-pt Likert Scale of importance was used, where 1 represented extremely important and 7 was not at all important.Q: “Onscale from1e7,where 1 is ‘extremely important’ and7 is “not at all important”, please rate the importance of eachof the following ethical and societal values in guidingHTA”.

a Wilcoxon Signed Ranks Test was used. Missing values were excluded.

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e108

documentation about how ethical and social issues are identifiedand reviewed in HTA (Lehoux & Williams-Jones, 2007; Potter,Avard, Entwistle, Kennedy, Chakraborty, McGuire et al., 2008). Inlight of the increasing difficulties HTA producers face whenaddressing value judgments into the HTA process (Gauvin et al.,2010), and the limited capacity to incorporate ethical analysis inHTA (Lehoux & Williams-Jones, 2007), it is imperative to developa framework of how to incorporate public input and values intoHTA. A first question is where public opinion and such valuejudgments should be incorporated e should they be introduced inthe evaluation or appraisal stages of HTA? Whereas evaluationinvolves the systematic gathering and summarizing of scientificevidence, appraisal falls in the realm of a political process, wheredecision-making constitutes the combination of the scientificevidence with values (ten Have, 2004). Some literature points tothe latter, suggesting that evidence produced by the evaluation aretypically interpreted within a framework of values and preferences(ten Have, 2004). One could questionwhether the appraisal processis too late in the process in which to incorporate public input andvalues. Previous work (Bombard, 2009), as well as the findings fromthis study, demonstrate that citizens can provide values-basedinput to inform both the early stages of the HTA process (e.g.,scoping or vignette stage) as well as the later stage of draftingrecommendations. Despite these developments, further research isneeded to comparatively assess the value added at each stage.

Arguably, there is also a need to evaluate the products of publicengagement processes, especially prior to their inclusion into HTAor appraisals. Admittedly, the fields of public engagement in HTA

Table 3Attitudes towards the importance of the top ethical and societal values identified in me

PR

n

Top Societal Values Identified:Evaluating how the technologies affect distribution of health care 11Ensuring that technologies are aligned with social values and arrangements 11Evaluating how the technologies relate to more general challenges

of modern medicine11

Ensuring that the users in the studies are representative of the usersthat will apply it in clinical practice

11

Top Ethical Values Identified:Ensuring that individual autonomy is respected 11Protecting the physician-patient relationship 11Ensuring that human dignity is respected 11Ensuring that basic human rights are respected 11

A7-pt Likert Scale of importance was used, where 1 represented extremely important anQ: “Onscale from1e7,where 1 is ‘extremely important’ and7 is “not at all important”, please

a Wilcoxon Signed Ranks Test was used. Missing values were excluded.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

and ethics in HTA are in their infancy and evaluation frameworksare only beginning to emerge in the former domain. Themajority ofpublic engagement evaluation frameworks outside of HTA havefocused on procedural aspects (Abelson et al., 2003; Daniels, 2000;Rowe & Frewer, 2000). Although recent work by De Vries et al.(2010) offers important advances in assessing the quality ofdemocratic deliberation, ultimately the effectiveness of any publicengagement process must also be judged in relation to theoutcomes achieved. In particular, understanding the impact of suchexercises on decision-making within HTA agencies will be helpfultoward any future endeavor aimed at the long-term incorporationof public engagement within HTA.

There are several caveats in the interpretation of our study. First,it is unknown to what degree Hofmann’s questions influenced orlimited the identification of the Panel’s values. The presentation ofonly one approach to considering ethical and social issues in HTAmay be akin “to choosing a single utopian moral perspective asanswering all the ethical dilemmas likely to arise in society”(Lehoux &Williams-Jones, 2007, pg.15). Related to this, through theprocess of distilling and re-organizing Hofmann’s list of 33 moralquestions to a more manageable list of 10 ethical questions and 10socially-relevant questions, we may have steered the panel’sdeliberations by introducing some restrictions on the range ofquestions presented and through the categorization process. Tooffset this, panel members were exposed to Hofmann’s article andfull set of moral questions and given the opportunity to add valuesto the list of those up for consideration and did so. Designers ofthese processes need to pay careful attention to the trade-offs

eting #5.

E-MEETING POST-MEETING P-VALUEa

Mean SD n Mean SD

1.73 1.27 10 1.90 1.45 0.412.45 1.63 10 2.50 1.27 1.002.00 1.48 10 2.20 1.55 0.32

1.64 1.21 10 2.10 1.54 0.10

1.73 1.49 10 1.60 1.58 0.562.45 1.51 10 2.40 1.65 0.321.91 1.51 10 2.00 1.49 1.001.91 1.45 10 2.10 1.60 0.32

d 7 was not at all important.rate the importance of eachof the following ethical and societal values in guidingHTA”.

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e10 9

between providing lists and guiding frameworks a priori andensuring that comprehensive efforts are made to solicit the public’sethical and social values toward health technologies.

Finally, the Panel reviewed five health technologies that focusedon diagnostic testing and population-screening modalities, whichwere at various stages in the review and appraisal process. Weacknowledge that these technologies may be limited in scope andnot representative of health technologies in general; and thus, it ispossible that the core values identified are not representative of thevalues at stake across any technology nor can findings be gener-alized to other jurisdictions. Further research is warranted toestablish core values across different jurisdictions and acrossamore expansive scope of technologies. Given the higher educationand income levels present in our Panel, it is possible that thesevalues may not represent minority groups or the general pop-ulation but rather citizens who may be more resourceful or expe-rience fewer challenges in accessing health care. Additional workcould explore the degree to which there is general acceptance ofthese values by a larger, representative sample.

Conclusion

The core values considered at stake across a variety of healthtechnologies included: universal access, choice and quality care.Ethical and social values should be considered at all stages of thehealth technology assessment process, and relevant values shouldbe considered on a case-by-case basis. Those designing processesfor eliciting ethical and social values to inform HTA and relatedpolicies need to carefully balance the principles of pragmatismwhile ensuring that quality public input is obtained. Sustained anddeliberative methods, like a citizens’ panel, offer a promisingapproach for eliciting ethical and social values into the HTA processbut these approaches require further study in relation to otherparticipatory methods as well as other methods for consideringethical and social values in HTA decision-making.

Acknowledgments

We are grateful to our panel members for generously sharingtheir time and insights with us. This project was led by Dr. JuliaAbelson at McMaster University with funding provided by theOntario Ministry of Health and Long-term Care through the HealthTechnology Evaluation Fund. Dr. Yvonne Bombard is supported bya Canadian Institutes of Health Research (CIHR) PostdoctoralFellowship and fellowships from the CIHR Strategic TrainingPrograms of ‘Public Health Policy’ and ‘Health Care, Technology andPlace’. We thank Lyn Sauberli for the administrative assistanceprovided throughout this project and the Centre for HealthEconomics and Policy Analysis at McMaster University for infra-structure support.

The sponsor had no involvement in study design; in thecollection, analysis and interpretation of data; in the writing of thearticle; and in the decision to submit it for publication.

Appendix. Supplementary data

Supplementary data associated with this article can be found, inthe online version, at doi:10.1016/j.socscimed.2011.04.017.

References

Abelson, J., Forest, P. G., Eyles, J., Smith, P., Martin, E., & Gauvin, F. P. (2003).Deliberations about deliberative methods: issues in the design and evaluationof public participation processes. Social Science & Medicine, 57(2), 239e251.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Abelson, J., Giacomini, M., Lehoux, P., & Gauvin, F. P. (2007). Bringing ‘the public’into health technology assessment and coverage policy decisions: from prin-ciples to practice. Health Policy, 82(1), 37e50.

Abelson, J., Miller, F. A., & Giacomini, M. (2009). What does it mean to trust a healthsystem? A qualitative study of Canadian health care values. Health Policy, 91(1),63e70.

Autti-Rämö, I., & Mäkelä, M. (2007). Ethical evaluation in health technologyassessment reports: an eclectic approach. International Journal of TechnologyAssessment in Health Care, 23(1), 1e8.

Banta, D. (2003). The development of health technology assessment. Health Policy,63(2), 121e132.

Benatar, D. (2006). Bioethics and health and human rights: a critical view. Journal ofMedical Ethics, 32(1), 17e20.

Bombard, Y. (2009). Public Engagement Pilot Study on Point- of-Care InternationalNormalized Ratio (INR) Monitoring Devices. In Medical Advisory SecretariatField Evaluations (Ed.).

Bridges, J. F., & Jones, C. (2007). Patient-based health technology assessment:a vision of the future. International Journal of Technology Assessment in HealthCare, 23(1), 30e35.

Burgess, M. M. (2004). Public consultation in ethics: an experiment in represen-tative ethics. Journal of Bioethical Inquiry, 1(1), 4e13.

Calnan, M. W. (1998). The patient’s perspective. International Journal of TechnologyAssessment in Health Care, 14(1), 24e34.

Chafe, R., Merali, F., Laupacis, A., Levinson, W., & Martin, D. (2010). Does the publicthink it is reasonable to wait for more evidence before funding innovativehealth technologies? The case of PET scanning in Ontario. International Journalof Technology Assessment in Health Care, 26(2), 192e197.

Childress, J. F., Faden, R. R., Gaare, R. D., Gostin, L. O., Kahn, J., Bonnie, R. J., et al.(2002). Public health ethics: mapping the terrain. Journal of Law, Medicine, andEthics, 30(2), 170e178.

Cleemput, I., Leys, M., Ramaekers, D., & Bonneux, L. (2006). Balancing evidence andpublic opinion in health technology assessments: the case of leukoreduction.International Journal of Technology Assessment in Health Care, 22(4), 403e407.

Coulter, A. (2004). Perspectives on health technology assessment: response fromthe patient’s perspective. International Journal of Technology Assessment inHealth Care, 20(1), 92e96.

Daniels, N. (2000). Accountability for reasonableness. BMJ, 321(7272), 1300e1301.De Vries, R., Stanczyk, A., Wall, I. F., Uhlmann, R., Damschroder, L. J., & Kim, S. Y.

(2010). Assessing the quality of democratic deliberation: a case study of publicdeliberation on the ethics of surrogate consent for research. Social Science &Medicine, 70(12), 1896e1903.

DeJean, D., Giacomini, M., Schwartz, L., & Miller, F. A. (2009). Ethics in Canadianhealth technology assessment: a descriptive review. International Journal ofTechnology Assessment in Health Care, 25(4), 463e469.

Dowlen, O. (2008). Sorted: Civic lotteries and the future of public participation.Toronto: MASS LBP.

European Network for Health Technology Assessment (EUnetHTA). (2007). HTA coremodel for medical and surgical interventions e First public draft (revised). Workpackage 4: Common Core HTA. European Network for Health TechnologyAssessment (EUnetHTA).

Gauvin, F. P., Abelson, J., Giacomini, M., Eyles, J., & Lavis, J. N. (2010). It all depends:conceptualizing public involvement in the context of health technologyassessment agencies. Social Science & Medicine, 70(10), 1518e1526.

Giacomini, M., Hurley, J., Gold, I., Smith, P., & Abelson, J. (2004). The policy analysisof ‘values talk’: lessons from Canadian health reform. Health Policy, 67(1),15e24.

Goven, J. (2008). Assessing genetic testing: who are the “lay experts”? Health Policy,85(1), 1e18.

Heitman, E. (1998). Ethical issues in technology assessment. Conceptual categoriesand procedural considerations. International Journal of Technology Assessment inHealth Care, 14(3), 544e566.

Heitman, E. (1999). Social and ethical aspects of in vitro fertilization. InternationalJournal of Technology Assessment in Health Care, 15(1), 22e35.

Hofmann, B. (2005). Toward a procedure for integrating moral issues in healthtechnology assessment. International Journal of Technology Assessment in HealthCare, 21(3), 312e318.

Hutcheon, P. D. (1972). Value theory: towards conceptual clarification. BritishJournal of Sociology, 23(2), 172e187.

International Network of Agencies for Health Technology Assessment (INAHTA)Ethics Working Group. (2005). Ways to handle ethical issues e Final report.Stockholm: INAHTA.

Johnson, A. P., Sikich, N. J., Evans, G., Evans, W., Giacomini, M., Glendining, M., et al.(2009). Health technology assessment: a comprehensive framework forevidence-based recommendations in Ontario. International Journal of Tech-nology Assessment in Health Care, 25(2), 141e150.

Karpowitz, C. F., & Mansbridge, J. (2005). Disagreement and consensus: the need fordynamic updating in public deliberation. Journal of Public Deliberation, 1(2),348e364.

Kass, N. E. (2001). An ethics framework for public health. American Journal of PublicHealth, 91(11), 1776e1782.

Kenny, N., & Giacomini, M. (2005). Wanted: a new ethics field for health policyanalysis. Health Care Anal, 13(4), 247e260.

Lehoux, P., & Williams-Jones, B. (2007). Mapping the integration of social andethical issues in health technology assessment. International Journal of Tech-nology Assessment in Health Care, 23(1), 9e16.

and social values in health technology assessment: A participatory04.017

Y. Bombard et al. / Social Science & Medicine xxx (2011) 1e1010

Morse, J. M. (1991). Approaches to qualitative-quantitative methodological trian-gulation. Nursing Research, 40(2), 120e123.

Mullen, C. (2008). Representation or reason: consulting the public on the ethics ofhealth policy. Health Care Anal, 16(4), 397e409.

National Institute for Health and Clinical Excellence. (2008). Social value judge-ments: Principles for the development of NICE guidance.

Office of Technology Assessment. (1976). Development of medical technology:Opportunities for assessment. Washington DC: U.S. Government Printing Office.

Pivik, J., Rode, E., & Ward, C. (2004). A consumer involvement model for healthtechnology assessment in Canada. Health Policy, 69(2), 253e268.

Potter, B. K., Avard, D., Entwistle, V., Kennedy, C., Chakraborty, P., McGuire, M., et al.(2008). Ethical, legal, and social issues in health technology assessment forPrenatal/Preconceptional and newborn screening: a Workshop report. PublicHealth Genomics, 12, 4e10.

Public Engagement Subcommittee of the Ontario Health Technology AdvisoryCommittee. (2007). Final report of the public engagement subcommittee. In O.H. T. A. C. Reports (Ed.). Toronto.

Reuzel, R. (2004). Interactive technology assessment of paediatric cochlearimplantation. Poiesis & Praxis, 2, 119e137.

Rokeach, M. (1973). The nature of human values. New York: Free Press.Rokeach, M. (1979). Understanding human values: Individual and societal. New York:

Free Press.Rowe, G., & Frewer, L. J. (2000). Public participation methods: a framework for

evaluation. Science, Technology and Human Values, 25(1), 3e29.

Please cite this article in press as: Bombard, Y., et al., Eliciting ethicalapproach, Social Science & Medicine (2011), doi:10.1016/j.socscimed.2011.

Saarni, S. I., Hofmann, B., Lampe, K., Luhmann, D., Mäkelä, M., Velasco-Garrido, M.,et al. (2008). Ethical analysis to improve decision-making on health technolo-gies. Bulletin of the World Health Organization, 86(8), 617e623.

Sandelowski, M. (2000). Whatever happened to qualitative description? Research inNursing and Health, 23(4), 334e340.

Sandelowski, M. (2010). What’s in a name? Qualitative description revisited.Research in Nursing and Health, 33(1), 77e84.

Scully, J. L., Banks, S., & Shakespeare, T. W. (2006). Chance, choice and control: laydebate on prenatal social sex selection. Social Science & Medicine, 63(1), 21e31.

Sorrenson, C., Drummond, M., Kristensen, F. B., & Busse, R. (2008). How can theimpact of health technology assessments be enhanced? Health systems and policyanalysis. World Health Organization.

Strauss, A. C. J. (1998). Basics of qualitative research: Techniques and procedures fordeveloping grounded theory. Thousand Oaks: Sage Publications.

ten Have, H. (2004). Ethical perspectives on health technology assessment. Inter-national Journal of Technology Assessment in Health Care, 20(1), 71e76.

Upshur, R. E. (2002). Principles for the justification of public health intervention.Canadian Journal of Public Health, 93(2), 101e103.

van der Wilt, G. J. (2004). Health technology assessment: trying to bring empiricaland ethical inquiry together. Poisesis Praxis, 2(2e3), 195e206.

Velasco, M., Perleth, M., Drummond, M., Gurtner, F., Jorgensen, T., Jovell, A., et al.(2002). Best practice in undertaking and reporting health technology assess-ments. Working group 4 report. International Journal of Technology Assessmentin Health Care, 18(2), 361e422.

and social values in health technology assessment: A participatory04.017