SOCIOLOGY 331
RESEARCH METHODS
Assignment 1:
Evaluation of a peer-reviewed article (25 points):
Due no later than
11:55p ET on Sunday of Week 3
Purpose: The purpose of exercise is to conduct
adetailed, critical evaluation of the research design, methods and
analysis of a study written up and published in a peer-reviewed
journal. Students will be using Wolfer’s (2007) critical questions
for evaluating written research to evaluate an article; these questions are
provided below.
Article
selection tips:
–
select an article from a peer-reviewed Sociology Journal
–
the article should clearly be written about a study the author conducted
o a meta-analysis of multiple studies is not
appropriate for this assignment
o secondary data analysis of an existing national
data set is OK, but be sure you understand what you are reading
Requirements:
Your evaluation should include
– All sections 1-5,
– 1 section from 6-10
(based on design in your article), and
– 1 section from 11-12
(based on analysis in your article).
This assignment does not have to be written in
essay format. You may organize it by section/chapter number, question
number with question text, and then your answer.
APA formatting should be used throughout.
Any time your paraphrase or directly quote a source (such as your article),
in-text citations should be used. A full APA-formatted reference should
be included at the beginning or end of assignment.
Disclaimer
Originality of attachments will be verified by Turnitin.
Both you and your instructor will receive the results.
1.Title (3 pts)
1)
Is the title specific enough to differentiate it from other related topics?
2)
Do subtitles, if present, provide important information regarding the research?
3)
Are the main variables expressed in the title?
4)
Are the terms in the title easily understood by most people?
5)
Does the title avoid any reference to the study’s results?
6)
Overall, is this a good title? Why or why not?
2. Ethical Evaluation (2.5 pts)
7)
Are the steps the researcher took to honor ethical responsibilities to
individuals clear? Are they appropriate? Are they enough?
8)
If there were any findings (based on your readings of tables or other means of
data presentation) that refuted the researcher’s hypothesis, did he address
these findings?
9)
If any results were unexpected, did the researcher discuss any explanations for
the unexpected effects?
10) Did the
researcher adequately acknowledge the limitations of the research?
11) Overall, has
the researcher adequately fulfilled his ethical obligations?
3. Literature Review (4 pts)
12) Is the
material presented in the literature review relevant to your research
interests?
13) Is the special
problem area identified in the first paragraph or two of the report?
14) Does the
researcher establish the importance of the research problem?
15) Has the
researcher been appropriately selective in deciding what studies to include in
the literature review?
16) Is the
research cited recent?
17) Is the
literature review critical?
18) Is the
researcher clear as to what is research, theory and opinion?
19) Overall, do
you think this is an adequate literature review? Why or why not?
4. Operationalization and Measurement (5.5 pts)
20) Is the
conceptualization suitably specific?
21) Are the
definitions productive?
22) How many
different dimensions are being measured at once?
23) Are the
various dimensions sufficient?
24) Are the actual
questions (or a sample of them) provided?
25) Is the
response format clear, or, when not already clear, does the researcher provide
information on the response format? Is there any information on
restrictions in respondents’ responses?
26) If the
researcher is using a published instrument, does he or she cite sources where
additional information can be found?
27) Has the
researcher avoided overstating the preciseness of the measurement?
28) Does the
researcher provide some measure of reliability? What type of reliability
is established? Do the measures indicate adequate reliability for your
purposes?
29) Does the
research provide some measure of validity? What measures of validity are
presented and are they adequate for your purposes?
30) Overall, is the
measurement appropriate and adequate given the research purpose?
5. Sample Strategy (3 pts)
31) Does the
research goal lend itself to generalization? Is the broad sampling method
appropriate for the research goal?
32) Does the
researcher provide information regarding the study population? The
sample?
33) Is the exact
sampling method (e.g. simple random, purposive) specified? Remember, it
is not sufficient for a researcher to simply state that a sample was selected
‘randomly.’
34) Is the sample
size sufficient, given the research goals, the degree of accuracy the
researcher desires, and the nature of the population studied? Given the
nature of the research, is the sample size sufficient?
35) If the
researcher uses a probability sample, does he or she generalize the findings to
the appropriate population? If the researcher uses a non-probability
sample, does he or she refrain from generalizing to a wider population?
36) Overall, is
the sampling appropriate?
Your evaluation should include one of the
following sections (6-10) (4 pts):
6. Experiments
37) Can you
identify a treatment variable that indicates that an experiment is the method
of observation?
38) How many
groups were studied?
a.
If there were two or more groups, did the researcher use random assignments
b.
If the researcher didnot use random assignment, did the researcher present evidence
that the groups were similar regarding key variables at the beginning of the
study?
39) Is the
treatment and any pre- or posttests described in sufficient detail that
facilitates replication?
40) Is deception
necessary?
a.
If so, is the deception within the parameters of the research topic?
b.
Have the participants been debriefed so they know the true nature of the study
(and can enact their right to privacy by declining to participate after the
fact?)
41) Based on the
description of treatment and experimental procedure, do you see any red flags
regarding ethical issues?
42) Did the
researcher use assistants?
a.
If so, did the researcher state that they were properly trained?
b.
If so, did the researcher specify any special measures to make sure that the
assistants administered the treatment properly?
43) Is the setting
natural or artificial (in a laboratory)?
a.
If it’s in a laboratory, does the researcher recognize that external validity
may be weak?
b.
If it’s in a natural setting, does the researcher recognize that there may be
some differences in the environments of the various groups?
c.
Overall, do you think the experimental design is sound?
7. Survey
44) Is the
research topic worded appropriately for survey research?
45) Did the
researcher specifically state which type of survey method was used?
46) Do the survey
questions adequately address the topic?
47) Are the survey
questions constructed correctly?
48) Did the
researcher provide any information about the response rate? Did the researcher
provide any information about follow-up mailings or other ways of increasing
response rate? What are the implications of the response rate?
49) Did the
researcher explain how he or she guaranteed anonymity or confidentiality?
50) Overall, is
the survey methodology effective and appropriate?
8. Field Research
51) Does the
research describe the selected site? Does the research provide some explanation
as to how that site was chosen?
52) Did the
researchers explain how they addressed gatekeepers?
53) Did the
researcher address how he developed field relations? If conflict arose,
did the researcher make any comment about how personal or research problems in
the field were addressed?
54) Did the
researcher adequately protect the identity of the respondents? Did the
researcher address other ethical considerations?
55) Did the
researcher describe, at least in passing, his method of note taking? Does
the method seem adequate?
56) In the
analysis, does the researcher present general patterns of behavior and support
those patterns with data such as quoted comments? Does the researcher use
quotes selectively?
57) Does the
researcher make any mention of issues of validity and/or reliability?
58) Overall, is
the research adequate?
9. Unobtrusive Measures
59) What is the
researcher’s research purpose or hypothesis? Is content analysis an
appropriate method of observation?
60) What are the
researcher’s units of analysis? What are the units of observation (if
they are different than the units of analysis)?
61) Is the
researcher studying a population or a sample of these units? If the
researcher is studying a sample, is it a probability sample? If so, was
it correctly drawn? If the researcher is not studying a population or a
probability sample, is he or she appropriately cautious about the nature of any
conclusions?
62) Does the
researcher identify the characteristics and level of content being
analyzed? Does the researcher explain how material is coded, especially
for issues of latent content?
63) Did the
researcher do any type of pretest with other coders to test for
reliability? Where they any tests for validity?
64) Are the
conclusions consistent with the units of analysis?
65) Are the
results clearly presented and the conclusions appropriate?
66) Generally, is
the method of observation done appropriately?
10. Evaluation Research
67) What is the
purpose of the evaluation presented?
68) Is the nature
of the program described in detail?
69) Are the goals
presented and can the goals that the author presents be evaluated?
70) What type of
observation method is used? Is it appropriate, given the real-life restrictions
of evaluation research?
71) Is a control
group used? If so, how has the researcher tried to show that it is equivalent
to the experimental group? If not, does the researcher adequately explain
its omission?
72) How are people
selected for program participation? Does this affect the interpretation of
findings, and, if so, does the researcher discuss this?
73) Are the
results clearly explained?
74) How does the
article address the other areas of evaluation discussed in earlier chapters?
Your evaluation should include one of the
following sections (3 pts):
11. Qualitative Analysis
75) Is the results
section a cohesive essay?
76) Does the
researcher connect the results to any general research questions or goals?
77) Is the
perspective of the results presentation appropriate? Does it match the research
technique?
78) Has the writer
presented enough examples to support the conclusions? Do the examples
make the readers ‘believe’ the researcher’s points?
79) Do you have
reason to believe that the presence of the researcher influenced the actions or
statements of other group members? If this is possible, has the researcher
addressed it in the research?
80) Especially in
field research (although this may be an issue to a lesser degree in other forms
of qualitative data gathering), does the researcher discuss how he or she
interacted with subjects in the field, what problems arose, and how the
researcher addressed them?
12. Quantitative Analysis
81) Is the results
section a cohesive essay with the important findings highlighted?
82) In the essay,
does the researcher tie the results to the research hypotheses or goals stated
in the introduction?
83) If there are
tables or graphs, are they clearly presented?
84) Does the
researcher present any descriptive statistics?
85) Are the
statistics appropriate for the level of measurement?
86) Are the
conclusions the researcher draws appropriate for the statistical information?
87) In the
discussion section, does the researcher briefly summarize the research
purposes, methodologies, and key findings (in a non-statistical manner)?
88) Does the
researcher acknowledge any methodological or statistical weaknesses?
89) Are the
implications of the research or suggestions for future research discussed?
90) Overall, is
the results section adequate?
91) Overall, is
the discussion section adequate?
