Toxicology Excellence for Risk Assessment (TERA) is an independent non-profit organization with a mission to protect public health through the best use of toxicity and exposure information in the development of human health risk assessments. As a non-profit organization, one of TERA’s programs is to provide peer consultation and peer review services to meet the needs of public and private sponsors. TERA organizes independent peer reviews and consultations, generally at the request of the authors or sponsors of a chemical assessment or other risk assessment work product.
The purpose of a peer consultation or review is to convene a group of expert peers to evaluate the scientific basis and appropriateness of the document and its conclusions. For example, the U.S. Environmental Protection Agency (EPA) Peer Review Handbook defines peer review as a documented critical review of a work product that is conducted by qualified individuals who are independent of those who performed the work, but are collectively equivalent in technical expertise (i.e., peers) to those who performed the work (EPA 2000). A peer review is an in-depth assessment of the assumptions, calculations, alternate interpretations, methodology, and conclusions of the document under review. A peer consultation, in contrast, is generally held at an earlier stage of document development. Panel members evaluate the scientific basis and appropriateness of the document and its conclusions, as well as provide the sponsors with recommendations for improvement or options to consider. An objective evaluation by independent experts with a variety of different viewpoints and perspectives is critical to the credibility of any peer consultation or peer review.
Generally the sponsor of the peer review or consultation pays for the direct costs of conducting the peer review meeting and for TERA’s labor costs to organize and convene the meeting. TERA’s responsibilities included identifying and recruiting scientists with relevant expertise, identifying and managing conflict of interest and bias issues, organizing and conducting the meeting, and drafting and finalizing the meeting report. TERA peer review panels generally volunteer their time. Peer consultation panels may or may not be offered an honorarium for their time.
TERA conducts its scientific peer reviews and consultations in accordance with U.S. EPA peer review procedures (as described in EPA 2000). In addition, TERA has developed its peer review and consultation program following principles highlighted by the American Industrial Health Council’s 1995 report Fundamental Scientific Peer Review Principals and utilizing approaches used by the National Academy of Sciences and EPA’s Science Advisory Board. The International Life Sciences Institute (ILSI) Policies and Procedures for its Model Peer Review Center of Excellence was based in part on TERA’s program.
Development of Charge
TERA reviews a draft of the document to be discussed in order to develop the Charge to Peer Reviewers/Consultants. TERA generally prepares the Charge in consultation with the sponsors and authors to insure that the review or consultation covers all the significant scientific issues and questions. EPA’s Peer Review Handbook describes the charge as a document that identifies recognized issues and invites comments. The Charge focuses the review by presenting specific questions and concerns for the reviewers to address and it invites general comments on the scientific and technical merits of the document. If appropriate, the Charge is based upon a standard format TERA uses that includes questions regarding literature completeness and interpretation of data and key decisions for hazard assessment and dose response. In addition, more specific questions relevant to the chemical or assessment are always included.
Selection of the Panel and Evaluation of Potential Conflict of Interest and Bias Issues
As indicated in EPA’s Peer Review Handbook, peer reviewers and consultants should be selected for both independence and scientific/technical expertise. Each panel member should have recognized technical expertise that bears on the subject matter under discussion. TERA strives to include a range of perspectives on each panel, including diverse professional affiliations (e.g., academic, consulting, environmental, government, and industry). The evaluation of real or perceived bias or conflict of interest is an important consideration and for both peer review and consultation panels and every effort is made to avoid conflicts of interest and biases that would prevent a panel member from giving an independent opinion on the subject.
TERA begins its panel search by reviewing the document or product and identifying the key areas of scientific expertise that are required to ensure that the panel will conduct a thorough and insightful review. TERA may ask the sponsors and authors (and others) to suggest types of expertise or nominate qualified individuals for consideration. However, TERA is solely responsible for identifying types of expertise and selecting the panel.
TERA examines the experience and credentials of any nominees, and independently identifies additional experts with expertise in the key areas. Sources for identifying appropriate potential panel members include TERA’s internal database of reviewers (which have been accepted by its Board of Trustees), Internet and literature searches, and professional contacts and referrals. TERA compiles a pool of candidate panelists with expertise in the needed areas and then narrows this list to those who are the most qualified scientifically, keeping in mind the goal of a balanced panel with a range of affiliations and expertise. Each candidate is then evaluated for potential conflict of interest and bias issues.
Identification and management of potential conflict of interest (COI) and bias issues are paramount to the success and credibility of any peer review or consultation composed of scientific experts. TERA utilizes a multi-step process to identify potential conflicts or biases. In initial conversations with each candidate, TERA discusses the nature of the review, the sponsor, and other interested parties and stakeholders. TERA asks the candidates questions regarding their work and relationships with these parties. Each potential panelist is given a copy of TERA’s COI policy statement and asked to complete a questionnaire to determine whether their involvement in certain activities could pose a conflict of interest or could create the appearance that the panelist might lack impartiality. This questionnaire covers employment history, financial aspects, and professional affiliations of the candidates. TERA staff carefully reviews these forms and discusses the answers with the candidates to determine whether conflicts of interest or unacceptable bias might exist.
TERA’s conflict of interest policy (see http://www.tera.org/Peer/COI.html) identifies the following situations as examples of those that could create a real or perceived conflict of interest.
- Working for an organization that sponsors or contributes to the document to be reviewed,
- Having direct personal financial investments benefiting from the outcome of the review, or
- Authoring or providing significant comments on the document.
The TERA COI policy also discusses bias. For these reviews, “bias” means a predisposition towards the subject matter under consideration that could influence the candidate’s viewpoint. Examples of bias would be situations in which a candidate:
- Has previously taken a public position on subjects to be discussed, or
- Is affiliated with an industry, governmental, public interest, or other group with a partiality regarding the subjects to be discussed.
Most scientists with technical expertise in areas relevant to a risk assessment document will have existing opinions about some of the subject matter, and therefore, may be considered to have some degree of bias. For some reviews or consultations, TERA may forward a short list of the candidates to the sponsors for their input regarding adequacy of credentials of panel members, or unidentified conflict of interest issues. In all cases, TERA is solely responsible for determining who will serve on the panel. TERA’s final selection of the panel members is based upon the candidates’ scientific experience and credentials, the overall need for coverage of the charge questions, conflict of interest and bias considerations, and the individuals’ interest and availability. Experts are asked to serve on the panel to provide their personal scientific opinions of the issues under discussion during the meeting; they do not serve as advocates of any particular position or represent any specific group or agency.
Preparation for the Peer Review or Consultation Meeting
TERA, as the convener of the meeting is responsible for all preparations for the review or consultation including travel and logistics, announcements, distribution of the review materials, and assisting the panel. TERA develops the agenda to make most effective use of the meeting time. TERA distributes the review package, which includes the document prepared by the Sponsor, key references, charge to reviewers, instructions to the panel, and logistics information. This package is sent to the panel about a month prior to the meeting. Sponsors agree to provide copies of any cited reference to panel members upon request.
Conduct of the Meeting
Each peer review or consultation meeting generally follows a standard TERA process, beginning with a close examination of the work product and supporting documentation by the panel prior to the meeting. TERA organizes the meeting to make best use of the time available to hear the opinions of the experts on the scientific positions and conclusions presented in the document.
At the start of the meeting, all panelists disclose and discuss their conflict of interest and bias issues. TERA believes transparency in these matters is important and therefore discusses these openly at the meeting, allowing panel members to question one another. These disclosures are also part of the public record through inclusion in the meeting report.
The chair then discusses the ground rules for the meeting. The discussion of issues at the meeting is limited to panel members. One or two authors or representatives are asked to sit at the table to answer panel questions. These representatives are also allowed to ask the panel members clarifying questions as needed. In order to avoid the appearance of undue influence on the panel, all parties are asked to refrain from discussing issues related to this review with panel members prior to the meeting or during the breaks unless a panel member initiates the discussion. Panel members are asked to summarize any substantive conversations for the rest of the panel and audience when the meeting reconvened after the break.
The meeting begins with the authors or sponsors making an initial short presentation describing the work product. The purpose of this presentation is to highlight the salient points and issues, and to give the panel the opportunity to ask clarifying questions of the authors. Technical comments from the public are then sought with the panel provided the opportunity to ask clarifying questions. The meeting Chair then leads the panel in discussions, covering the issues and questions from the Charge to the peer reviewers. Individual panelists are asked to share their opinions and defend them with scientific data and analysis. If a panelist finds the work product lacking, he or she will be asked to explicitly identify what additional work would be needed to resolve the deficiency or error. For peer reviews, the panel attempts to achieve consensus on the key points and Charge questions. Consensus for the purposes of these meetings is defined as the panel’s collective opinion or general agreement (Webster's Pocket Dictionary, new revised edition, 1997). Consensus positions that are unanimous are stated as "unanimous consensus.” If consensus is not achieved, the meeting report will discuss the differing opinions. If unanimous consensus is not achieved, the meeting report will also show the differing opinions, but will also allow a minority opinion to be appended upon request. In this way, the full range of opinions of the panel is part of the public record. For peer consultations, panelists will share their personal opinions and critically question one another, but consensus is not necessarily sought. Rather, all individual opinions of the panel members are recorded
Opportunities for Public Involvement
Members of the public are generally invited to attend the independent peer reviews and consultations organized by TERA. It is important that the entire process be transparent so that interested parties can judge the independence and scientific credibility of the review or consultation. The public may be given the opportunity to provide brief oral and written technical comments on the work product for the panel’s consideration. To facilitate meaningful public participation, TERA makes available the meeting agenda, logistics, Charge, and sometimes the work product to the public via the Internet. For each meeting, TERA provides a meeting web page (accessed through www.tera.org/peer) where the key information is provided in a timely fashion for all interested parties. This is also where the final report from the consultation or review is made available to the public.
Senior TERA scientist(s) take notes of the discussions and prepare a draft meeting report that summarizes the panel’s discussions, conclusions, and recommendations. This report is not a transcript of the meeting; rather it summarizes the key discussions and issues. The draft report is reviewed and approved by the panel. The sponsors are also allowed to review the draft report, but must limit their comments to matters of clarity and completeness regarding their presentations and their statements from the meeting. The meeting report contains copies of the sponsor presentation slides, a list of attendees, panel biographical sketches and COI/bias disclosures, handouts from the meeting, and public comments. The meeting reports are made available to the public on a specific meeting web page at TERA’s web site.