I–Title, Author, Location:
Goal: Identify the likely credibility of the material and the likelihood that the paper contains new, true and important information.
Title: Two general questions:
1) Is it new?
a) Check PubMed for similar articles.
b) Does the article’s title indicate anything new in terms of results, concepts or methodology?
2) Is the topic important?
a) Does the article’s title indicate the paper addresses a significant problem or concern?
b) Would the conclusion (if indicated in the title and true) of the article influence (potentially at least) treatment or clinical or basic research?
c) Does the article’s title indicate that a significant amount of new information is presented?
Location and Credibility
1) Determine relative credibility of the source:
Refereed journal— if so, standing of the journal in its field (for example its impact factor) , association with learned society.
Unrefereed or unknown:
Promotional Material:
2) Is the location appropriate for its intended audience? That is
Date of Publication and Currency:
1) Is the information current or has more recent information on this topic been published subsequently?
2) If the article is > than one year old, has it been cited by others?
Authors:
1) Do the authors have a track record of research in this field—(PubMed search)
2) Is their work cited (Google Scholar)
3) Is their any evidence of possible conflict of interest (e.g. sponsored research)?
Abstract:
Goals:The reader’s goals are a) to determine whether the paper is worth reading and if so 2) to identify issues of possible concern and an agenda of active reading to determine that the conclusions are warranted. The authors’ goals are to summarize the study in terms of its pupose, methods, results and conclusions, and maximize the importance of the findings.
1) Are the objectives of the study clearly stated?
2) Identify the general study design (e.g. observational, correlational, experimental) with its associated strengths and weaknesses (e.g threats to validity), and the particular study design (e.g. naturalistic enquiry, case-control, randomized controlled trial see Glossary,).
3) Identify possible sources of bias in literature review, subject selection, outcome measures, procedures and interpretation. Assess in a preliminary manner, whether the identified biases are likely to affect the conclusions (see Glossary).
4) Determine if it is a positive or negative (e.g. no difference between groups or no relationship between groups) results paper: a) if positive are their any obvious alternative hypotheses to explain the results? b) if negative, did the experiment have sufficient statistical power? (see CT pp see CCCT — sample size calculator).
5) Identify the methods used and whether they are standard (in which case what are their significant weaknesses) or novel (in which case, how well documented and validated).
6) Identify the conclusion and assuming competence in execution, assess whether the work presented, satisfies the Assertability Question i.e. What evidence would it take to convince me of the truth of this conclusion? Or looked at another way, what are the appropriate standards for research in this area and does the article achieve them?
7) Do the results and conclusions appear to be new, true and important?
8) Key issues: identify the key issues that challenge the soundness of the conclusions.
Your Personal Relevance
Having carefully considered the abstract, determine what purpose would be served by your reading this article?
1) Do you have enough knowledge to understand the article?
2) What level of certainty do you require in accepting the conclusions – are your future actions, such as patient treatment or investigations ,likely to be influenced by it?
3) If you feel the article is likely to have sufficient quality to justify the time spent analyzing it, proceed to a detailed analysis, section by section below.
Introduction:
Goals. The authors goals are: 1) to present the context of the study (a bird’s eye view) , briefly describing previous findings that are to be challenged or developed, 2) state the hypotheses or assumptions underlying the study 3) catch the readers’ attention and direct them down the path that follows 4) frequently to outline the main findings or conclusions of the paper.
The readers goals are: 1) to determine that the theoretical framework is balanced and free of empty rhetoric 2) to determine if the study is exploratory or hypothesis driven 3) to determine if there is a specific question being asked that is directly related to the hypothesis.
1) Balanced Framework?. Are the references used in the Introduction recent and reflect the overall work in the area being investigated? For example, are contrary results or authors who hold different theoretical views excluded? (Note a PubMed search informs this point.)
2) Hypothesis-Driven?. Is a specific answerable question being posed and do the conclusions relate directly to this question? Or has a broad exploratory study been undertaken and conclusions reached by data dredging?
3) Are the questions being asked novel or will the results only confirm previous findings?
4) Are the questions being asked dealing with important issues?
Materials and Methods:
Purpose: To give the reader enough information to judge the validity of the results and repeat the experiments.
Authors’ Goals: Justify the design of the study and mention the limitations on the conclusions imposed by the methods used, while providing sufficient detail to enable others to repeat the experiments.
Readers’ Goals: Identify limitations in the materials and methods that affect the conclusions.
General Issues:
1) Definitions and Classification: Are terms used precisely? Are the operational definitions appropriate? Are groups classified appropriately and unambiguously? Are the categories non-overlapping but still inclusive (i.e. all members of the population fit somewhere)?
2) Is the study design appropriate for the question being asked. For example, a cross-sectional design would not normally be appropriate for proving causation.
3) Study design and possible threats to validity. Note the threats to validity that you have already identified in your perusal of the abstract are possible threats, some may not be important to the conclusion in a particular investigation. Assess the extent to which they affect the conclusion
4) Adequacy of Measurement.
Choice of Units (for example, direct measures vs. ratios or combined indices)
Resolution
Accuracy.
Validity
Reliability.
Type of scale (nominal, ordinal, interval, ratio)
5) For clinical studies –how were cases recruited and are they representative of the condition being studied (see Glossary for Sackett’s catalogue of biases, see also CT pp 15-6,88,135,250-2).
6) For experimental studies- why was the particular model system chosen and are its results likely to inform clinical practice or clarify mechanisms?
7) Biases. Are the measurements biased in any way? See Glossary for Sackett’s catalogue of possible biases including measurement bias.
8) Appropriate statistical analysis: (calculator)
a) Was an appropriate test used?
b) Was a statistician (either as co-author or listed in the acknowledgements) involved in the analysis of the data? (Of particular importance if an obscure test was used.)
c) Were the comparisons planned in advance or was retrospective subgroup analysis used that might lead to spuriously significant effects.
d) If pairing or matching was done in the statistical analysis, was it done appropriately?
e) If groups are to be compared over time, were they comparable at baseline (i.e. at time zero) and if not was any adjustment in either analysis or interpretation made?
f) Was there appropriate justification provided for using a one-tailed test?
g) If a statistically significant result was found, was the size of the effect or correlation sufficiently large to be important?
h) Did the authors employ any of Greenhalgh’s ?0 ways to cheat on statistical tests?(BMJ 315;422 1997)
i) throw all data into a computer and report as significant any relation where p.05
ii) if baseline favours intervention group do not adjust
iii) do not test for a normal distribution
iv) ignore withdrawals and non-compliers
v) always plot one set of data against another to get an r value and when significant state it proves causation
vi) use outliers to advantage, eliminate if they hurt retain if they help
vii) omit confidence intervals that include zero effect
viii) stop the trial when the results are significant
ix) if results uninteresting, check out subgroups to see if they behaved differently
x) if you don’t get the result you want, try another test
9. Are materials and methods described in such detail that the investigation could be repeated?
10. Are up-to-date techniques used?
11. Issues Specific to Investigational Approaches:
A: For a brief guide to the issues for specific designs proceed to the approach given below.
CONSORT (CLINICAL TRIALS) http://www.consort-statement.org/
ORION (outbreak or intervention of a nosocomial organism) http://www.bsac.org.uk/_db/_documents/ORION_Checklist.doc
QUORUM (meta analysis) http://www.consort-statement.org/mod_product/uploads/QUOROM%20Statement%201999.pdf
MOOSE: (Meta-analysis of observational studies in epidemiology) http://jama.ama-assn.org/cgi/content/abstract/283/15/2008
STARD (DIAGNOSTIC TESTS) http://www.stard-statement.org/
STRICTA (ACUPUNCTURE) http://www.stricta.info/journals.htm
Strobe: (observational studies in epidemiology) http://www.strobe-statement.org/
TREND (nonrandomized designs) http://www.trend-statement.org/asp/comments.asp
MIBBI (minimum information for biological and biomedical investigations includes guidelines for such approaches as microarrays, cellular assays etc.) http://www.mibbi.org/index.php/Main_Page
B: Observation/Description:
a) Assess the effects of threats to validity for the design in question and determine the degree to which they are important. Two examples follow
i) e.g. in naturalistic observation there is a concern that the personal biases of the investigator influence data analysis and interpretation. Techniques to minimize this concern include explicit systematic examination of observations by such techniques as triangulation (determining that different sources support the same interpretation), saturation (recording of observations until nothing new is being uncovered), checks on an understanding or perception with members of the group being observed, explicit discussion of personal biases, a record of the investigator’s thinking and coding decisions that can be audited by others, and peer review of the findings (see CT 199-203).
ii) one shot case study: in such a before/after study history, maturation, selection, regression to the mean and mortality are potential threats to validity.(see CT pp 203-4, p303). Sometimes historical controls are used i.e. comparisons are made to results that have been obtained previously with other interventions or no treatment at all. Opinion is divided on their usefulness, but it appears that even if the direction of treatment outcome effect is correct, sizes of effects are often overestimated.
b) Assess the degree to which the observational methods altered the observations.
c) How were the observations selected, and did the selection affect the conclusions? For example was sampling random or purposive?
d) Were the observations detailed, systematic, and recorded immediately?
e) To what extent could the observations been influenced by the investigator expectations?
B. Correlational (see CT pp 207-15)
a) For all correlational studies it must be asked whether there are confounding variables that cloud the interpretation? Were adjustments either in interpretation or statistical analysis made for confounders?
b) Does the paper meet the Feinstein’s scientific standards for correlational studies.
i) high quality data
ii) stipulated research hypothesis
iii) wellspedified cohort
iv) avoidance of detection bias
v) accurate determination of exposure to the agent in question.
c) Cross Sectional
i)Do the investigators inappropriately interpret cross-sectional correlations as proving causation?
ii) Have the investigators used an appropriate sampling scheme so that the subjects surveyed accurately represent the population to which the results are being applied ?
d) In ecologic studies, are the correlations observed in populations likely to be found when considered on an individual subject basis.
e) Case-control study:
i) Are the cases representative of the condition in the general population? (highly screened samples such as those admitted to a specialized centre may not be.)
ii) The cases should not be selected for study on account of their exposure to the agent in question as this may generate a spurious association?
iii) Are the cases carefully defined? Are the cases homogeneous or are disparate conditions lumped together?
iv) is the exposure confounded with other agents that might influence the outcome?
i) Are the controls in a case-control study:
-Representative of the general population?
– Or do they exhibit relationships with the factors under study in a way that might influence the results?
-Was there a good match between the controls and the cases?
ii) Is the information (on exposure to some agent or condition), which may, for example in retrospective studies, have been collected some considerable time in the past, complete and accurate? Could there be recall bias?
f) Cohort prospective design:
i) If a cohort prospective design of an intervention is used,
– was there good control of the therapy or exposure, as well as the observations?
-Did the subjects remain blind to their treatment?
-Are statistical issues, such as the effects of drop-outs and randomization, addressed properly?
ii) if the outcome of interest is rare was the sample size and time of observations sufficient to observe a reasonable number of events so that a fair test of the effect of the exposure could be obtained?
f) Cohort retrospective design
i) Is the information on exposures reasonably comprehensive or is there reason to think that exposure to other possible relevant agents could also be involved (e.g. lung cancer incidence in smokers who are asbestos miners)
C) Experiment (see CT 217-290):
a) Does the investigator deal appropriately with controlled, uncontrolled and uncontrollable variables.
b) Are there adequate controls including postive, negative, and active controls?
c) Is there adequate measurement and statistics (absence of or correction for systematic error), sufficient precision and calculation of uncertainty.
d)A wide range of validity?
e) Is the experiment sufficiently simple in design that it could be executed competently?
f) Does the investigation use innovative tactics?
g) What measures has the investigator taken to optimize the effects and do these measures impinge on the range of validity.
h) For clinical trials, were the issues of outcome measures, blinding, compliance, subject recruitment and loss dealt with appropriately.
i) Was high quality data obtained?
j) Does the publication meet the CONSORT guidelines (see Glossary)
k) Are the following important biologic variables considered or controlled: genetics, sex, activity, emotionality, microbiologic pattern, environment and diet?
D) Meta-analysis:
a) does the publication meet the QUORUM criteria (see Glossary)
8) Overall, were the observational tools and procedures employed the best available ?adequate? Inadequate?
9) Does the study meet the reporting criteria for its type of design or field of investigation.
CONSORT (CLINICAL TRIALS) http://www.consort-statement.org/
ORION (outbreak or intervention of a nosocomial organism) http://www.bsac.org.uk/_db/_documents/ORION_Checklist.doc
QUORUM (meta analysis) http://www.consort-statement.org/mod_product/uploads/QUOROM%20Statement%201999.pdf
MOOSE: (Meta-analysis of observational studies in epidemiology) http://jama.ama-assn.org/cgi/content/abstract/283/15/2008
STARD (DIAGNOSTIC TESTS) http://www.stard-statement.org/
STRICTA (ACUPUNCTURE) http://www.stricta.info/journals.htm
Strobe: (observational studies in epidemiology) http://www.strobe-statement.org/
TREND (nonrandomized designs) http://www.trend-statement.org/asp/comments.asp
MIBBI (minimum information for biological and biomedical investigations includes guidelines for such approaches as microarrays, cellular assays etc.) http://www.mibbi.org/index.php/Main_Page
10. Are there any innovations in design, measurement, treatments or other methodology that might be useful in other investigations?
RESULTS:
Purpose: To present data in a clear objective fashion so that the reader can judge the soundness of the conclusions.
Authors’ apparent objective: To make the data look as convincing as possible
Readers’ objective:
1) to assess the strength of the data underlying the conclusions
2) to consider other ways of presenting the data that may or may not lead to different conclusions
Key issues include:
1) Technical quality
Example: do photomicrographs actually demonstrate what they purport to demonstrate? Are they in focus, and cover a suitable area?
Example:
Radiology: dental radiographs must be technically sound enough to determine the following (Macdonald D:
a) The shade of the lesion? Is it radiolucent or radiopaque or a mixture of both (invariably a radiopacity within a radiolucency).
b) Is the lesion multiple or single? – if multiple- likely represent systemic cause, whereas the single lesion is likely to have a local cause/s
c) Is the margin of the lesion well or poorly defined? – if poorly defined consider a malignant or inflammatory cause, well-defined suggests a benign process.
d) What is the lesion’s relative location: dental (or mandibular) canal (or the image of the hard palate for the upper jaw)- if it lies above the inferior dental canal ( or below the image of the hard palate) then it lies in the alveolar or tooth bearing process. This suggest that the lesion is of odontogenic origin.?br>
2) Numerical quality: e.g. reporting number of experimental units, variation, partition of variation, standardization
3) What was lost when the data were processed. (e.g. reporting of ratios typically obscures absolute values of measurements and may suppress appearance of variation see CT CH 14) )
3) Consistency: with other studies and within the study. For example do the numbers add up, do data in different tables and figures reconcile?
4) Was data that was either collected or should have been collected left out?
5) Were the effects reported statistically significant?
6) Were the effects or relationships reported large enough to be important.
7) Tables: are there irregularities in the data hidden in tables?
8) Do the figures/illustrations conform to Tufte’s criteria and Cleveland’s hierarchy (see Glossary and CT Chapter 13)
9) Do the authors engage in the following deceptive or attempted deceptive practices: biased selection and/or maipulation of data, persuading with “pap” providing biased context? (CT CH13).
10) Did the authors use innovative ways of presenting the data and, in particular do these methods allow relationships to be depicted that might otherwise be obscured?
DISCUSSION & CONCLUSIONS:
Authors’ Intent: To convince the reader of the truth of the conclusions.
Typical Reader Concerns:
1) Does the study answer the questions posed in the Introduction or at least make some points that are relevant to the original hypothesis.
2) Are conclusions based on all available relevant evidence or by a biased selection of evidence? Are published data that conflict with the conclusions included and adequately explained?
3) Does the equation Valid Logic + True Premises = Sound Conclusions apply to this paper?
4) Are the limitations of the evidence or conclusions discussed?
5) Are speculations and opinion clearly identified as such?
6) By using an argumentation map, can you demonstrate the relationship between the conclusion and supporting evidence presented in the paper?
7) Are any of the conclusions unsupported by data?
8) Are there any plausible alternative hypotheses to explain the data? Common approaches to criticizing hypotheses (see CT pp77-8) include:
i) challenge to definitions and terms
ii) different conclusions from the same data
iii) different antecedent or cause
iv) interference because of ohe factors
v) irrelevant reasons or relationships
vi) too much/too little (e.g. fluoride arguments)
vii) factor ignored
viii) counter examples
ix) saving revision
9) In the instance of “negative results” i.e. no effect or relationship being found, what was the sensitivity of the experiment? For example was there sufficient statistical power to see meaningful effects?
10) Was the paper free from common fallacies (e.g. irrelevant evidence, post-hoc etc, biased statistics etc. (see CH 8).
10) Are there any hidden assumptions or suppressed premises in the authors’ reasoning?
11) Are there ways that the paper could be improved?
TAKE HOME MESSAGE
1) Do you agree in general with the conclusions of the paper or do you reject the authors conclusions outright because of some fatal flaw in any aspect of the paper?
2) If you agree in general with the conclusions do you think the conclusions should be modified on account of deficiencies in some aspects or limitations of the study.
3) State in your own words the take-home message of the paper.
POSITIVES: What Did You Learn?
Identify the positive features of the paper, such as sound conclusions, good or novel methodology, references that you were unaware of, convincing rhetoric such as good justification for the topic area or rationale for the methods used in the investigation, statistical procedures, indicators of what is current level of evidence for publication in the journal, ideas the article engendered in you for future (hopefully improved) studies.
