Health Evidence Bulletins - Wales (logo)

Stages of the methodology

Roles and Responsibilities

STAGE 1: Defining The Task

Identify subject areas for searching by examining the existing bulletin (if one already exists) and on advice from the Internal Review Group and other interested parties (health professionals and members of the public) Team Leader/Information Resources Manager
Decide deadlines, personnel requirements for the critical appraisal process (Stage 3) and suitable members for the Internal Review Group (Stage 5). Agree membership of the Internal Review members with the Steering Group.
[Each Appraisal Team should adopt a multidisciplinary approach and be aware of the information needs of medical, nursing and therapy professions as well as purchasers and consumers]
[Members of the Internal Review Group should be acknowledged experts in their field. A multidisciplinary approach should be adopted to ensure coverage of all subject areas and viewpoints. The Steering Group should be informed once membership is finalised]
Team Leader/Steering Group
Offer a training session on the Project Methodology and the use of the Critical Appraisal Checklists to each Critical Appraisal Team if required. Inf. Res. Manager


STAGE 2: Searching For Information

Search for information in the subject areas identified in Stage 1 using appropriate sources from checklist (Appendix 1) and utilising master searches where available (Appendix 2). Carry out searches for each subject area in a stepped fashion for Types I to V evidence in a strict order (see Appendix 3 for explanation of evidence types). Inf. Res. Manager
Carry out searches for a subject from the date given on the relevant chapter of an earlier bulletin if one exists. For new subject areas, search five-ten years initially depending on the amount of information in a given subject area. Search earlier dates if no type I or II evidence is found. Inf. Res. Manager
Include English and non-English articles. Where the non-English abstracts seem to indicate type I or II evidence, arrange the translation of full-text documents. Inf. Res. Manager
Store reference details, remove duplicates and provide output in a suitable format using reference management software. Inf. Res. Manager
Provide results from database searches to the Team Leader at each step for an initial grading of the information. Include Cochrane reviews and other key documents in full text at this stage. Inf. Res. Manager
Annotate search results as follows and return search to Information Assistant:
/ Obtain document in full text
NR Reject (not relevant)
ID Reject (inappropriate/inadequate research design)
OTH Reject (other: add reason briefly)
Team Leader/
Appraisal Team
Acquire full-text publications in accordance with Team Leader’s instructions. Information Assistant


STAGE 3: Grading The Evidence

For each full-text document supplied, attach an Evidence Form (Appendix 4) to provide the bibliographic details and indicate the type of evidence. Supply a copyright declaration with each document if required, to be signed by recipient and returned to the Project Office. Inf. Res. Manager/
Information Assistant
Organise the critical appraisal of papers using the appropriate Quality Appraisal Checklist as a guide (Appendices 5-9). Team Leader/
Critical Appraisal Team
Co-ordinate the critical appraisal process and communicate with, and motivate, the critical appraisal team. Team Leader
Decide on the evidence to be included in the first draft of the Bulletin. Team Leader/CA Team
Store evidence to be included and critical appraisal checklists. Evidence excluded (and appraisal checklists) should also be retained in case of query. Team Leader/or
Inf. Resources Manager by request.


STAGE 4: Assigning The Evidence To Statements

Update existing statements as appropriate. Team Leader/CA Team
Write new statements where evidence is found which is relevant to the subject area, and of important current interest, but was not covered in an existing bulletin. Team Leader/CA Team
The Team Leader and Information Resources Manager to liaise closely, to ensure that, once evidence has been assigned to statements covering a particular subject, no other types of evidence will be sought. Team Leader/
Inf. Res. Manager
Prepare the first draft of the Bulletin which should include a matrix of statements and the evidence which supports them (see Appendix 10).

The statements should include (where appropriate):

  1. the target population (e.g. age, sex);
  2. quantitative information (e.g. Number Needed to Treat, % improvement);
  3. a Health Gain notation (indicating the potential benefit to health, see Appendix 11).

The evidence should be:

  1. cited according to the Vancouver System
  2. accompanied by an evidence type and any other relevant information about the research
  3. (See Appendix 11 for an example )

Team Leader
Identify areas of little or no evidence, conflicting evidence or where evidence is of poor quality in order to:
either pursue the search further
or recommend a systematic review
or make a recommendation for further research
Team Leader



Submit the first draft of the Bulletin to the Internal Review Group asking for the following responses:
  1. Do they agree with the statements and the supporting evidence?
  2. Are they aware of any better evidence than that quoted?
  3. If they do not agree, why not?
  4. What evidence do they have?
  5. Are they aware of any examples of good practice?
  6. Any other comments.
Team Leader
As an ideal, convene the Internal Review Group to a meeting for this purpose. Team Leader
If this is not possible, the Team Leader should gather and expedite responses, and liaise with reviewers over queries. Team Leader
Carry out further searching, appraisal and evaluation as necessary. Team Leader/
Inf. Res. Manager
Draw up a final draft of the bulletin in consultation with the Steering Group. Team Leader/
Inf. Res. Manager/
Management. Group


Submit bulletin to Steering Group for approval and forwarding to External Review. Team Leader/
Inf. Res. Manager/
Management Group



Select External Reviewers of high standing (using suggestions from the Welsh and English Offices of Research & Development) to appraise final Bulletins and the conduct of the Project. Steering Group; Wales Office of Research & Development; English Offices of Research & Development
Co-ordinate the External Review process. Project Director
Give approval for dissemination of the Bulletins. Steering Group


Complete an action plan for the dissemination of each Bulletin, clearly covering:
  1. Distribution list (in primary, secondary, tertiary settings);
  2. World-wide publication;
  3. Relevant talks and conferences
Steering Group
Carry out an ongoing evaluation programme to inform the development of both text and electronic versions of the documents.  Information Resources Manager/Steering Group





Team Leader
Liaises with Information Resources Manager and Assistant
Establishes a Local Appraisal Team for the Protocol
Co-ordinates the activities of the Appraisal Team to ensure consistency, timeliness and quality
Decides on evidence for inclusion
Identifies and reports areas of poor evidence
Draws up matrix of statements and supporting evidence
Writes up preliminary reports for review by Internal Review Group
Co-ordinates the assessment of these reports
Writes up final report to submit to Steering Group for External Review

Information Resources Manager supported by Information Assistant

Methodology Home Top