The Mysteries of the AAPOR Code… What is it and what happens if someone violates it? Mary E. Losch & Stephen Blumberg AAPOR Standards Chair & Associate.

The Mysteries of the AAPOR Code…

What is it and what happens if someone violates it?

Mary E. Losch & Stephen Blumberg

AAPOR Standards Chair & Associate Standards Chair

Presentation to DC AAPOR

March 2009

AAPOR Code of Professional Ethics and Practice

http://www.aapor.org/aaporcodeofethics

Schedule of Proceduresfor Code Violations

http://www.aapor.org/scheduleofproceduresforcodeviolations

http://www.aapor.org/aaporcodeofethics

http://www.aapor.org/scheduleofproceduresforcodeviolations

I. Principles of Professional Practice in the Conduct of Our Work

A. We shall exercise due care in developing research designs and survey instruments, and in collecting, processing, and analyzing data, taking all reasonable steps to assure the reliability and validity of results.

1. We shall recommend and employ only those tools and methods of analysis that, in our professional judgment, are well suited to the research problem at hand.

2. We shall not knowingly select research tools and methods of analysis that yield misleading conclusions.

3. We shall not knowingly make interpretations of research results that are inconsistent with the data available, nor shall we tacitly permit such interpretations.

4. We shall not knowingly imply that interpretations should be accorded greater confidence than the data actually warrant.

I. Principles of Professional Practice in the Conduct of Our Work

B. We shall describe our methods and findings accurately and in appropriate detail in all research reports, adhering to the standards for minimal disclosure specified in Section III.

C. If any of our work becomes the subject of a formal investigation of an alleged violation of this Code, undertaken with the approval of the AAPOR Executive Council, we shall provide additional information on the survey in such detail that a fellow survey practitioner would be able to conduct a professional evaluation of the survey.

II. Principles of Professional Responsibility

in Our Dealings With People A. The Public:

1. When preparing a report for public release we shall ensure that the findings are a balanced and accurate portrayal of the survey results.

2. If we become aware of the appearance in public of serious inaccuracies or distortions regarding our research, we shall publicly disclose what is required to correct these inaccuracies or distortions, including, as appropriate, a statement to the public media, legislative body, regulatory agency, or other appropriate group, to which the inaccuracies or distortions were presented.

3. We shall inform those for whom we conduct publicly released surveys that AAPOR standards require members to release minimal information about such surveys, and we shall make all reasonable efforts to encourage clients to subscribe to our standards of minimal disclosure for their releases.


in Our Dealings With People B. Clients or Sponsors:

1. When undertaking work for a private client, we shall hold confidential all proprietary information obtained about the client and about the conduct and findings of the research undertaken for the client, except when the dissemination of the information is expressly authorized by the client, or when disclosure becomes necessary under the terms of Section I-C or II-A of this Code.

2. We shall be mindful of the limitations of our techniques and capabilities and shall accept only those research assignments that we can reasonably expect to accomplish within these limitations.


in Our Dealings With People C. The Profession:

1. We recognize our responsibility to the science of survey research to disseminate as freely as possible the ideas and findings that emerge from our research.

2. We shall not cite our membership in the Association as evidence of professional competence, since the Association does not so certify any persons or organizations.


in Our Dealings With People D. The Respondent:

1. We shall avoid practices or methods that may harm, humiliate, or seriously mislead survey respondents.

2. We shall respect respondents' concerns about their privacy.

3. Aside from the decennial census and a few other surveys, participation in surveys is voluntary. We shall provide all persons selected for inclusion with a description of the survey sufficient to permit them to make an informed and free decision about participation.


in Our Dealings With People D. The Respondent

4. We shall not misrepresent our research or conduct other activities (such as sales, fund raising, or political campaigning) under the guise of conducting research.

5. Unless the respondent waives confidentiality for specified uses, we shall hold as privileged and confidential all information that might identify a respondent with his or her responses. We also shall not disclose or use the names of respondents for non-research purposes unless the respondents grant us permission to do so.

6. We understand that the use of our survey results in a legal proceeding does not relieve us of our ethical obligation to keep confidential all respondent identifiable information or lessen the importance of respondent anonymity.

III. Standards for Minimal Disclosure

Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted. At a minimum, the following items should be disclosed.

1. Who sponsored the survey, and who conducted it. 2. The exact wording of questions asked, including the text of

any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response.

3. A definition of the population under study, and a description of the sampling frame used to identify this population.

III. Standards for Minimal Disclosure

4. A description of the sample design, giving a clear indication of the method by which the respondents were selected by the researcher, or whether the respondents were entirely self-selected.

5. Sample sizes and, where appropriate, eligibility criteria, screening procedures, and response rates computed according to AAPOR Standard Definitions. At a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed.

6. A discussion of the precision of the findings, including estimates of sampling error, and a description of any weighting or estimating procedures used.

7. Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.

8. Method, location, and dates of data collection.

What happens if there is a violation?

Schedule of ProceduresLast Revised -- January 2006

1. Any individual, whether an AAPOR member or not, may submit a written complaint concerning a possible violation of the AAPOR Code of Professional Ethics and Practices to the Standards Chair.

1a. Complaints must be made in writing on an AAPOR Standards Complaint Form and must include

(i) a statement describing the alleged action or violation,

(ii) the section of the Code alleged to be violated,

(iii) the date or dates when the alleged violation occurred,

(iv) the name(s) of the individual(s), firm(s) or organization(s) alleged to be in violation, and

(v) the original signature of the complainant along with his or her contact information, including telephone number, postal address, and e-mail address.

Schedule of Procedures

1b. Any organization, firm, individual, or group of individuals may be named in such a complaint, whether or not it or they are members of AAPOR.

1c. The written complaint shall be accompanied by all available relevant evidence.

1d. At the discretion of the Standards Chair, any duly appointed member of the Standards Committee may be given confidential access to a received complaint, unless that Standards Committee member, or his or her organization, is the source or subject of that complaint. If the Standards Chair and Associate Standards Chair are thereby excluded, the Council shall appoint a Chair pro tem for that particular complaint.


1e. The complaint form and all related materials prepared for Council, the Standards Committee, and the Evaluation Committee (if one is appointed as provided in section 4 below) will be kept confidential, and all Council, Standards Committee, and Evaluation Committee discussions related to the specifics of the complaint will be treated as privileged communication.

1f. The procedures hereafter described will apply to all such written complaints received by the Standards Chair.


2. Within forty-five (45) days of receipt of a complaint, the Standards Chair shall decide, with the concurrence of the Associate Standards Chair or one other member of the Standards Committee, whether or not any official action is warranted. The Standards Chair may request assistance from the source or subject of the initial complaint, or use any other channels deemed helpful in recommending a course of action to Council.


2a. If no official action is to be recommended by the Standards Chair, he or she shall prepare a brief written report to Council, including the original complaint form and setting out the reason(s) why no action is indicated.

(1) If Council agrees with this opinion at its next regularly scheduled meeting, or after being polled by phone or e-mail, the source of the complaint shall be promptly notified as to the final disposition of the complaint.

(2) If Council does not agree with the Standards Chair’s decision, Council may call for clarification or direct the initiation of a full complaint review and investigation, as specified in sections 2b, 2c, and 3 below.

Schedule of Procedures2b. The Standards Chair may be directed by Council to seek

additional information and clarification about the alleged Code violation. In pursuit of this objective, the Standards Chair may request assistance from the source or subject of the initial complaint, or use any other channels deemed helpful in recommending a course of action to Council. The Standards Chair shall report on any additional information within thirty days of such a directive from Council.

2c. The Standards Chair may recommend and a Council majority approve of (or independently direct) the initiation of a full complaint review and investigation. In determining this course of action, Council will consider all relevant information, including the severity of the alleged violation(s) and facts gathered by the Standards Chair, including (if elicited) the stance initially taken by the subject(s) of the complaint.


3. If Council, by a majority of those voting, votes for a complaint review and investigation, the organization, firm or individual(s) who are the subject of the complaint shall be notified by the Standards Chair in writing within fourteen (14) days of Council action. Such notification will fully stipulate the nature of the complaint and shall include relevant portions of the Code and a copy of this Schedule of Procedures.


4. The investigation will be conducted by an ad hoc committee (hereafter termed Evaluation Committee), appointed by the Standards Chair subject to the approval of the Council.

4a. The Evaluation Committee will consist of no fewer than three nor more than five AAPOR members, one of whom Council will name as Chair. Neither the Standards Chair nor any other Council members may serve on this committee.

4b. The identity of the members of the Evaluation Committee will be known only to Council and to each other.

4c. No member of AAPOR shall serve on an Evaluation Committee in a matter where he or she or his or her organization is the source or subject of a particular complaint, or where his or her service could otherwise represent a potential conflict of interest in relation to the source or subject.


5. Within twenty-one (21) days of appointment, the Chair of the Evaluation Committee will receive all pertinent materials on the case that are held by the Standards Committee, and the Standards Chair will, in writing, request the subject(s) of the complaint to provide any other materials deemed necessary by the Evaluation Committee, or materials and statements which the subject(s) deem necessary or relevant to a fair investigation.


5a. Copies of all such materials received, including the original complaint and the notification to the subject(s), will be sent to each member of the Evaluation Committee.

5b. Any member of the Evaluation Committee may request the Standards Chair to seek further information or clarification of existing information.

5c. In conducting its affairs, the use of electronic conferencing techniques (phone, e-mail) by the Evaluation Committee, as needed, will be funded by AAPOR.

5d. The Committee may elicit and consider any relevant evidence.


6. If the subject(s) of the complaint fails to respond to the notice provided under paragraph 3 or fails to forward all material requested under paragraph 5 or 5b within forty-five (45) days of the notice or request, the Evaluation Committee will proceed, using the evidence at hand. If, however, the subject(s) of the complaint formally requests of the Standards Chair an extension of time, the Standards Chair may grant an extension in writing, upon determination that the extension is reasonable and necessary to compile all requested materials.


7. Within thirty (30) days of receiving the requested materials (or following the 45-day waiting period), the Evaluation Committee will complete a complaint review and investigation, reach a determination about the complaint, and will submit a written report to the Standards Chair. The complaint review and investigation will seek to determine the precise nature of the conduct which is subject to criticism and whether there is any factual basis to support the alleged misconduct. The Evaluation Committee must determine whether the information obtained may reasonably be interpreted to constitute a violation of AAPOR’s Code of Ethics and Practices, which reflects generally accepted professional practices. The committee’s report will describe the steps taken in review of the complaint, include the determination and a specific recommendation for Council action, and will conclude with one or more motions.

Schedule of Procedures7a. Council action to be recommended by the Evaluation

Committee may consist of, but is not limited to, any of the following:

(1) If AAPOR members or firms with AAPOR members as principals are subjects of the complaint:

a. Public or private exoneration;

b. Public or private censure;

c. Suspension or termination of membership;

d. Termination of investigation, with no further action.

(2) If non-members or firms without AAPOR members as principals are subjects of the complaint:

a. Public or private exoneration;

b. Public or private censure;

c. Termination of investigation, with no further action.

Schedule of Procedures7b. Before recommending any censure or altering of membership, the

Committee should consider the following criteria:

(1) The degree of severity of the violation(s);

(2) The subject’s history regarding other possible violations;

(3) The impact of the violation(s) on the public opinion and survey research profession;

(4) The impact of any proposed sanction(s) on the subject’s ability to successfully participate in public opinion and survey research in the future.

7c. Notwithstanding other actions taken, the Council may direct that a case study be prepared and publicized for the benefit of AAPOR members and other constituencies, with due regard given to the need to maintain confidentiality of the subject(s), information sources, and methods of investigation.

Schedule of Procedures8a. The Standards Chair will forward the Evaluation Committee’s report

to Council within fourteen (14) days of receipt, together with such views, in writing, as he or she may have. At its next regularly scheduled meeting thereafter:

(1) The Council (with at least two-thirds of its members voting) shall immediately move to accept, reject, or modify the report of the Evaluation Committee or of the Standards Chair.

Schedule of Procedures8a[1] a. If the Council’s decision exonerates the subject of the

complaint, the Standards Chair will promptly notify the subject(s) and no further Council action is required.

b. If the Council’s decision does not exonerate the subject, the Standards Chair will send notice of the Council’s decision to the subject by registered mail within fourteen (14) days. The Standards Chair’s letter shall include written notification advising the subject

(i) of the specific allegations and charges;

(ii) that, within thirty (30) days of receipt he or she may request a hearing before the full Council; and

(iii) that he or she will be given the right at said hearing to confront evidence and to refute all charges.


8a[1]c. The Council may also decide to request modification to the report of the Evaluation Committee. If such a request is made by Council, the Evaluation Committee has thirty (30) days to respond to such request. The revised report is then resubmitted for consideration, following the steps outlined in 8.1.


8a[2] Within thirty (30) days of receipt of notice of Council’s decision (section 8a[1]b), the subject of the complaint may request, in writing, a hearing before the full Council. Such a hearing shall be held at a time and place scheduled by the Council. The President shall control the procedure at the hearing, including presentation of testimony and submission of written materials, but shall not be bound by legal rules of evidence or procedures. Before the close of the hearing, the Council should consider the criteria in 7b above.


8b. Immediately following the close of the hearing - or, if no hearing is requested within the 30-day waiting period, at a regularly or specially convened meeting - the Council will make its decision (with at least two-thirds of its members voting), to accept, reject, or modify the judgment and recommendation of the Evaluation Committee or the Standards Chair.

8c. This decision will constitute the final action to be taken by the Council on behalf of AAPOR, except as allowed in section 7c. The subject of the complaint and the source of the original complaint will be notified in writing of the Council’s decision within fourteen (14) days by the President of AAPOR, who also will be responsible for implementing the terms of the decision.

The recent public case…• An allegation of violation of code was

received in March 2008

• Following the procedures, a review by Standards & Council resulted in a formal investigation by an evaluation committee

• The committee reviewed the evidence, made formal requests to the subject of the complaint for elements of minimal disclosure

The specific requests related to AAPOR’s finding of violation of minimum disclosure were as follows:

• The survey sponsor(s) and sources of funding for the survey.

• A copy of the original questionnaire or survey script used in the 2006 survey, in all languages into which it was translated.

• The consent statement or explanation of the survey purpose.

• A full description of the sample selection process, including any written instructions or materials from interviewer training about sample selection procedures.

• A summary of the disposition of all sample cases.


• How were streets selected? How were the starting street, and the starting household, selected? Once the starting point was selected, how were interviewers instructed to proceed (e.g., when they came to an intersection)? How were houses and respondents chosen at housing units?

• The survey description says that, “The interview team were given the responsibility and authority to change to an alternate location if they perceived the level of insecurity or risk to be unacceptable.” In how many clusters did the team change location, and what were the reasons for the changes?


• The survey description says that, “Empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations.” Were such cases included in the number of not-at-home and refusal cases counted in each cluster?

Dr. Burnham’s response was as follows:

• “This study was carried out using standard demographic and household survey methods.”

• “The methods we employed for this study were set out in the Lancet paper reporting our findings (Lancet, 2006;368:1421-28). The dataset from the study was released some time ago.”

The Outcome

• In accordance with AAPOR’s procedures, the evaluation committee completed its work and recommended public censure which was accepted by Council

• The public censure was manifested in the press release of February 4th, 2009

• Then all hell broke loose… ;-)

Reactions• The media – especially international – were interested in the action

• Some bloggers began flogging – “You are not the boss of him… “

• ASA was notably supportive

• A number of AAPOR members were delighted

• A number of AAPOR members seemed convinced that this Council was out of control and following no known guidelines… See second bullet above… “You are not the survey police”

Updates

• Dr. Burnham has not publicly responded to the AAPOR censure to our knowledge – has stated only that JHU instructed him not to respond

• JHU recently announced that the IRB had sanctioned Dr. Burnham for the same study – they have suspended Dr. Burnham's privileges to serve as a principal investigator on projects involving human subjects research for 5 years

• The specific details of the minimal disclosure request from AAPOR to Dr. Burnham (provided earlier in the slides) is now posted on the AAPOR website

Questions?

The Mysteries of the AAPOR Code… What is it and what happens if someone violates it? Mary E. Losch & Stephen Blumberg AAPOR Standards Chair & Associate.

Documents