Purpose: The purpose
of exercise is to conduct a detailed, critical evaluation of the
research design, methods and analysis of a study written up and published in a
peer-reviewed journal. Students will be using Wolfer’s (2007)
critical questions for evaluating written research to evaluate an article;
these questions are provided below.
Article
selection tips:
–
select an article from a peer-reviewed Sociology
Journal
–
the article should clearly be written about a
study the author conducted
o a meta-analysis of multiple studies is not appropriate for
this assignment
o secondary data analysis of an existing national data set is
OK, but be sure you understand what you are reading
Requirements:
Your evaluation should include
–
All sections 1-5,
–
1 section from 6-10
(based on design in your article), and
–
1 section from 11-12
(based on analysis in your article).
This assignment does not have to be written in essay
format. You may organize it by section/chapter number, question number
with question text, and then your answer.Include the questions so you are sure to respond to each
question and not miss any!
APA formatting should be used throughout. Any time
your paraphrase or directly quote a source (such as your article), in-text
citations should be used. A full APA-formatted reference should be
included at the beginning or end of assignment.
Disclaimer
Originality of attachments will be verified by
Turnitin. Both you and your instructor will receive the results.
General Reminder: For each response, please
provide at least 1-3 sentences to explain or describe, rather than just a ‘yes’
or a ‘no’ answer — this will strengthen your review, and provide more a
meaningful content to this assignment. These questions are to help guide you in
reviewing this peer-reviewed journal article.
1.Title (3 pts) (What’s the title?)
1)
Is the title specific enough to differentiate it from other related
topics?
2)
Do subtitles, if present, provide important information regarding the
research?
3)
Are the main variables expressed in the title?
4) Are
the terms in the title easily understood by most people?
5)
Does the title avoid any reference to the study’s results?
6)
Overall, is this a good title? Why or why not?
2. Ethical Evaluation (2.5 pts)
7)Are
the steps the researcher took to honor ethical responsibilities to individuals
clear? Are they appropriate? Are they enough?(Hint: IRB and Human Subject protection)
8)If
there were any findings (based on your readings of tables or other means of data
presentation) that refuted the researcher’s hypothesis, did he address these
findings?
9)If
any results were unexpected, did the researcher discuss any explanations for
the unexpected effects?
10)
Did the researcher adequately acknowledge the limitations of the
research?
11)
Overall, has the researcher adequately fulfilled his ethical obligations?
3. Literature Review (4 pts)
12)
Is the material presented in the literature review relevant to your
research interests?
13)
Is the special problem area identified in the first paragraph or two of
the report?
14)
Does the researcher establish the importance of the research problem?
15)
Has the researcher been appropriately selective in deciding what studies
to include in the literature review?
16)
Is the research cited recent?(Hint:
Look at the article’s ‘references’ — did they cite other research that are
more recent or are these older research?)
17)
Is the literature review critical? (Hint: Why or why not?)
18)
Is the researcher clear as to what is research, theory and opinion?(Hint: Why or why not?)
19)
Overall, do you think this is an adequate literature review? Why or
why not?
4. Operationalization and Measurement (5.5 pts)
20)
Is the conceptualization suitably specific?
21)
Are the definitions productive?
22)
How many different dimensions are being measured at once?
23)
Are the various dimensions sufficient?
24)
Are the actual questions (or a sample of them) provided?
25)
Is the response format clear, or, when not already clear, does the
researcher provide information on the response format? Is there any
information on restrictions in respondents’ responses?
26)
If the researcher is using a published instrument, does he or she cite
sources where additional information can be found?
27)
Has the researcher avoided overstating the preciseness of the
measurement?
28)
Does the researcher provide some measure of reliability? What type
of reliability is established? Do the measures indicate adequate
reliability for your purposes?
29) Does
the research provide some measure of validity? What measures of validity
are presented and are they adequate for your purposes?
30)
Overall, is the measurement appropriate and adequate given the research
purpose?
5. Sample Strategy (3 pts)
31) Does
the research goal lend itself to generalization? Is the broad sampling method
appropriate for the research goal?
32)
Does the researcher provide information regarding the study population?
The sample?(Hint: What are they?)
33)
Is the exact sampling method (e.g. simple random, purposive) specified?
Remember, it is not sufficient for a researcher to simply state that a
sample was selected ‘randomly.’(Hint: Provide
further explanation what method was used)
34)
Is the sample size sufficient, given the research goals, the degree of
accuracy the researcher desires, and the nature of the population studied?
Given the nature of the research, is the sample size sufficient?
35)
If the researcher uses a probability sample, does he or she generalize
the findings to the appropriate population? If the researcher uses a
non-probability sample, does he or she refrain from generalizing to a wider
population?
36)
Overall, is the sampling appropriate?
Your evaluation should include only one of the
following sections (pick one from 6 – 10)depending on the type of research method that was used in
that article (4 pts):
6. Experiments
37)
Can you identify a treatment variable that indicates that an experiment
is the method of observation?
38)
How many groups were studied?
a.
If there were two or more groups, did the researcher use random
assignments
b.
If the researcher did not use random assignment, did the
researcher present evidence that the groups were similar regarding key
variables at the beginning of the study?
39)
Is the treatment and any pre- or posttests described in sufficient detail
that facilitates replication?
40)
Is deception necessary?
a.
If so, is the deception within the parameters of the research
topic?
b.
Have the participants been debriefed so they know the true nature of the
study (and can enact their right to privacy by declining to participate after the
fact?)
41)
Based on the description of treatment and experimental procedure, do you
see any red flags regarding ethical issues?
42)
Did the researcher use assistants?
a.
If so, did the researcher state that they were properly trained?
b.
If so, did the researcher specify any special measures to make sure that
the assistants administered the treatment properly?
43)
Is the setting natural or artificial (in a laboratory)?
a.
If it’s in a laboratory, does the researcher recognize that
external validity may be weak?
b.
If it’s in a natural setting, does the researcher recognize that there
may be some differences in the environments of the various groups?
c.
Overall, do you think the experimental design is sound?
7. Survey
44)
Is the research topic worded appropriately for survey research?
45)
Did the researcher specifically state which type of survey method was
used?(Hint: what is it?)
46)
Do the survey questions adequately address the topic?
47)
Are the survey questions constructed correctly?(Hint: What are they like?)
48)
Did the researcher provide any information about the response rate? Did
the researcher provide any information about follow-up mailings or other ways
of increasing response rate? What are the implications of the response rate?
49)
Did the researcher explain how he or she guaranteed anonymity or
confidentiality?(Hint: How was it done?)
50)
Overall, is the survey methodology effective and appropriate?
8. Field Research
51)
Does the research describe the selected site? Does the research provide
some explanation as to how that site was chosen?
52)
Did the researchers explain how they addressed gatekeepers?(Hint: who are the gatekeepers for their study, if any?)
53)
Did the researcher address how he developed field relations? If
conflict arose, did the researcher make any comment about how personal or
research problems in the field were addressed?
54)
Did the researcher adequately protect the identity of the respondents?
Did the researcher address other ethical considerations?(Hint: How?)
55)
Did the researcher describe, at least in passing, his method of note
taking? Does the method seem adequate?
56)
In the analysis, does the researcher present general patterns of behavior
and support those patterns with data such as quoted comments? Does the
researcher use quotes selectively?
57)
Does the researcher make any mention of issues of validity and/or
reliability?(Hint: what are they?)
58)
Overall, is the research adequate? (Hint: why or why not?)
9. Unobtrusive Measures
59)
What is the researcher’s research purpose or hypothesis? Is content
analysis an appropriate method of observation?
60)
What are the researcher’s units of analysis? What are the units of
observation (if they are different than the units of analysis)?
61)
Is the researcher studying a population or a sample of these units?
If the researcher is studying a sample, is it a probability sample?
If so, was it correctly drawn? If the researcher is not studying a
population or a probability sample, is he or she appropriately cautious about
the nature of any conclusions?(Hint: don’t
guess, refer to the textbook as you explain the type of sample of this paper)
62)
Does the researcher identify the characteristics and level of content
being analyzed? Does the researcher explain how material is coded,
especially for issues of latent content?
63)
Did the researcher do any type of pre-test with other coders to test for
reliability? Were they any tests for validity?
64)
Are the conclusions consistent with the units of analysis?
65)
Are the results clearly presented and the conclusions appropriate?
66)
Generally, is the method of observation done appropriately?(Hint: why or why not?)
10. Evaluation Research
67)
What is the purpose of the evaluation presented?
68)
Is the nature of the program described in detail?
69)
Are the goals presented and can the goals that the author presents be
evaluated?(Hint: explain what the goals are, and why or
why not they can be presented)
70)
What type of observation method is used? Is it appropriate, given the
real-life restrictions of evaluation research?
71)
Is a control group used? If so, how has the researcher tried to show that
it is equivalent to the experimental group? If not, does the researcher
adequately explain its omission?
72)
How are people selected for program participation? Does this affect the
interpretation of findings, and, if so, does the researcher discuss this?
73)
Are the results clearly explained?(Hint: Why or why not?)
74) How
does the article address the other areas of evaluation discussed in earlier
chapters?
Your evaluation should include one of the following
sections (either 11 or 12) (3 pts):
11. Qualitative Analysis(Hint: Choose THIS if it is not does not mention statistics.
Common for observation, field research, interviews, focus groups – see more
about this in Babbie, 2011, pp. 24-25)
75)
Is the results section a cohesive essay?
76)
Does the researcher connect the results to any general research questions
or goals?
77)
Is the perspective of the results presentation appropriate? Does it match
the research technique?(Hint: provide
the justification)
78)
Has the writer presented enough examples to support the conclusions?
Do the examples make the readers ‘believe’ the researcher’s points?(Hint: why or why not?)
79)
Do you have reason to believe that the presence of the researcher
influenced the actions or statements of other group members? If this is
possible, has the researcher addressed it in the research?
80)
Especially in field research (although this may be an issue to a lesser
degree in other forms of qualitative data gathering), does the researcher
discuss how he or she interacted with subjects in the field, what problems
arose, and how the researcher addressed them?
12. Quantitative Analysis(Hint: Choose this if the author speaks of some kind of
instruments used, such as surveys, questionnaires, statistics, etc. Read more
in Babbie, 2011, pp. 24-25)
81)
Is the results section a cohesive essay with the important findings
highlighted?
82)
In the essay, does the researcher tie the results to the research
hypotheses or goals stated in the introduction?
83)
If there are tables or graphs, are they clearly presented?
84)
Does the researcher present any descriptive statistics?
85)
Are the statistics appropriate for the level of measurement?(Hint: Explain why or why not?)
86)
Are the conclusions the researcher draws appropriate for the statistical
information?
87)
In the discussion section, does the researcher briefly summarize the
research purposes, methodologies, and key findings (in a non-statistical
manner)?
88)
Does the researcher acknowledge any methodological or statistical
weaknesses?(Hint: What are they?)
89)
Are the implications of the research or suggestions for future research
discussed?
90)
Overall, is the results section adequate?
91)
Overall, is the discussion section adequate?
Remember
to include your Name and Title (cover) for this assignment, cite all resources,
and include the Reference section (following the APA format).
