532 CHAPTER 21 • PREPARING A RESEARCH REPORT
report, whether qualitative or quantitative. You are work is a delight. Write boldly and then say it
encouraged to examine and compare the reports again—better.
reprinted in previous chapters to see their differ- ■ Writing is an exercise in learning about
ences. Furthermore, when you are writing your re- your own work. Writing, then editing, then
port, you should look through journals pertinent to rewriting, then editing again, clarifies thoughts
your study to view the sections, level of detail, and into a coherent package.
types of results commonly reported to determine ■ Editing: Even a gem needs to be mined
the appropriate format for your report. roughly, cut ruthlessly, then buffed.
■ Nobody knows your work better than you do.
GUIDELINES FOR WRITING You’ll be surprised how much better you know
A RESEARCH REPORT it when you’ve discussed it with your computer
a few times.
In the spirit of sharing tips for successful writing, ■ Write without consideration for grammar,
we offer some tips in Figure 21.1 for what not to do syntax, or punctuation. Just write. Sometimes
once you finally get settled down to work. Activities editing is an avoidance technique.
such as those listed can eat up precious writing ■ Write at the same time every day, at a time
time. You may want to write down your own avoid- when you know you won’t be disturbed.
ance list and stick it next to wherever it is that you ■ Write about your research as though you’re
write. Check it occasionally, but get the behaviors sending an e-mail to a friend. Pretend your
out of your system. friend needs it explained simply.
When you’re clear on what not to do, focus on Writing is nothing more than putting thought
the following suggestions for what you should do to paper, yet many writers hold irrational beliefs
when writing a research report. about the task or even insist on rituals that must
be completed if writing sessions are to be success-
■ Look for progress, not perfection. ful. An informal survey of friends and colleagues
■ Write whatever comes to mind. Then go back yielded examples of such rituals—one suggested
that writing can occur only between the hours of
and hunt for what you are really trying to 7:00 AM and 12:00 PM, another that writing can be
say—it’s there. done only in longhand, using a blue pen and white
■ Have you ever thought to yourself, “I wish legal pad, and yet another that the house must be
I had done that differently?” With writing, clean before writing can start.
you can do it differently; editing your own
FIGURE 21.1 • Geoff’s tips for being able to avoid writing
• Think about all the things at school that I need to do before tomorrow.
• Scan my desk to see if someone has left me a note about a meeting, sports practice, birthday
party that I need to go to NOW.
• Check my voicemail.
• Check my e-mail.
• Check my checkbook to see if it is balanced.
• Call my wife/child/colleague/friend/enemy to see what they are doing.
• Walk down the hallway to see if I can find someone to talk to.
• Dream about winning the lottery.
• Make an appointment to see my dentist.
Source: Mills, Geoffrey, Action Research: A Guide for the Teacher Researcher, 4th Edition, © 2011, p. 180.
Reprinted by permission of Pearson Education, Inc., Upper Saddle River, NJ.
CHAPTER 21 • PREPARING A RESEARCH REPORT 533
There is no easy way around the pragmatic With these general guidelines for writing in
issue of time: Writing takes time, and we never mind, we next turn to some specifics about writ-
have enough time to do all that we have to do, pro- ing a research report. Probably the foremost rule of
fessionally and personally. The best advice we can research report writing is that the writer should try
offer is to make writing part of your professional to relate aspects of the study in a manner that accu-
life and responsibility. A study is not over until the rately reflects what was done and what was found.
report has been completed, and the time you will Although the style of reporting may vary for quantita-
need for writing should be included in the time tive and qualitative studies, the focus in all instances
schedule you create when planning your research should be on providing accurate description for the
(see Task 3 Example, p. 125). We know of no other reader. For example, in quantitative reports personal
way, besides attacking personal and family time, to pronouns such as I and we are usually avoided. In
get our writing done. In the short term, our loved contrast, in qualitative reports the researcher often
ones will put up with the “I need to stay home and adopts a more personal tone and shares the words
get this writing (grading, lesson planning, studying, of the participants. Such stylistic differences do not
etc.) done. You go ahead and enjoy the movie, din- alter the need for accurate reporting.
ner, picnic, hike, river rafting, skiing. . . .” We can
all fill in the blank based on our lives as students The research report should be written in a clear,
and teachers. Writing, however, need not wait un- simple, straightforward style that reflects scholarship.
til the rest of the study is complete. While you are You do not have to be boring, just concise. In other
collecting data, you can profitably use spare time words, convey what you want to convey and do it in
to begin revising or refining the introduction and an efficient way by avoiding jargon and using simple
method sections of your report, and after all the language. For example, instead of saying, “The popu-
data are analyzed, you can turn your attention to lation comprised all students who matriculated for
the final sections. the fall semester at Egghead University,” it would
be better to say, “The population was all students
The major guideline for all stages of writing enrolled for the fall semester at Egghead Univer-
is to begin with an outline. Developing an outline sity.” Obviously the report should contain correct
involves identifying and ordering major topics and spelling, grammatical construction, and punctuation.
then differentiating each major heading into logical Your computer probably has a spelling and grammar
subheadings. The time spent working on an outline checker. Use it, and when necessary, consult a dic-
is well worth it—it is much easier to reorganize an tionary. It is also a good idea to have someone you
outline that is not quite right than to reorganize a know, someone who is perhaps stronger in these
document written in paragraph form—and your areas, review your manuscript and indicate errors.
report will be more organized and logical if you
think through the sequence before you actually The final report should be proofread carefully
write anything. Of course, however, your first draft at least twice. Reading the report silently to yourself
should not be your last; two or three revisions of will usually be sufficient to identify major errors.
each section may be needed. Remember, writing If you have a willing listener, however, reading the
inevitably uncovers issues or activities that must manuscript out loud often helps you to identify
be rethought. Each time you read a section, you grammatical or constructional errors. Sometimes
should seek ways to improve its organization or sentences do not make nearly as much sense when
clarity, and asking others to review your report will you hear them as when you write them, and your
reveal areas in need of rethinking or rewording that listener will frequently be helpful in bringing to your
you have not noticed. attention sections that are unclear. Reading the re-
port backwards, last sentence first, will also help you
Fortunately, once you begin writing, it becomes to identify poorly constructed or unclear sentences.
easier to continue writing. You may even think of
a little reward system, if you are somewhat extrin- FORMAT AND STYLE
sically motivated. For example, after dedicating
himself to some writing time, one of your authors Format refers to the general pattern of organization
(Geoff) treats himself to a run, time with family or and arrangement of the report. Research reports
friends, or something sweet (you know, some sugar generally follow a format that parallels the steps
to help with the fatigue!).
534 CHAPTER 21 • PREPARING A RESEARCH REPORT
involved in conducting a study, although formats Style manuals also specify how to cite references
may vary in terms of which sections are included and use footnotes. In APA style, for example,
and how they are titled. For example, one format authors of cited references are usually referred to
may call for a section titled Discussion and another in the main body of the report by last name only;
format may require two separate sections, one ti- first names, initials, and titles are not given. Instead
tled Summary or Conclusions and another titled of writing, “Professor Dudley Q. McStrudle (2002)
Recommendations, but all formats require a sec- concluded . . . ,” researchers write, “McStrudle
tion in which the results of the study are discussed (2002) concluded . . . .” A reference page at the end
and interpreted. All research reports also include of the manuscript includes additional information
a condensed description of the study, whether it about the author and the source of the information
be a summary of a dissertation or an abstract of a cited.
journal article.
Check with your advisor about the style used in
Style refers to the rules of grammar, spelling, your institution before beginning writing—rearranging
capitalization, and punctuation followed in prepar- a format after the fact is tedious and time-consuming.
ing the report. Most colleges, universities, and pro- It is also very helpful to study several reports that
fessional journals require the use of a specific style, have been written in the style used by your institution.
either a style they have developed or that in a pub- For example, look at existing dissertations, especially
lished style manual. Perhaps the most frequently those directed by your advisor, to get an idea of format
cited manual used by educational researchers is the and what is expected (an institution may, in fact, be
Publication Manual of the American Psychological using a combination of styles and formats). If you are
Association, also called the APA manual (currently not bound by any particular format and style system,
in its sixth edition). we recommend that you acquire and study a copy of
the APA manual either in the full publication manual
Although different style manuals emphasize or “official pocket style guide.” Like other aspects of
different rules of writing, several rules are com- the research project, there are technology tools to help
mon to most manuals. Use of abbreviations and you manage your citations that will save you an enor-
contractions, for instance, is generally discour- mous amount of time when it comes to preparing your
aged in formal writing. For example, “the Ameri- research report. A few of these tools are outlined for
can Psychological Assn.” (which may appear in an you in the Digital Research Tools for the 21st Century
early draft) should be replaced by “the American feature.
Psychological Association.” Instead of shouldn’t,
researchers write should not. Exceptions to the FORMATTING THESES
abbreviation rule include commonly used and un- AND DISSERTATIONS
derstood abbreviations (such as IQ and GPA) and
abbreviations defined by the researcher to promote Although specifics will vary considerably, most
clarity, simplify presentation, or reduce repetition. research reports prepared for a degree require-
If the same sequence of words is used repeatedly, ment follow the same general format. Figure 21.2
the researcher often defines an abbreviation in presents an outline of the typical contents of such
parentheses when first using the sequence and a report. As the figure indicates, theses and dis-
thereafter uses only the abbreviation. Most style sertations include a set of fairly standard prelimi-
manuals address the treatment of numbers as well. nary pages, components that directly parallel the
One convention is to spell out a number that comes research process, and supplementary information,
at the start of a sentence (e.g., “Six schools were which is included in appendixes. A report on a
contacted . . .”). Another common convention is quantitative study and one for a qualitative study
to use words if the number is nine or less (e.g., have similar contents except that the method sec-
“a total of five lists . . .”) and to use Arabic numer- tion in the report of the qualitative study empha-
als if the number is greater than nine (e.g., “a total sizes the description and selection of the research
of 500 questionnaires was sent”). These guidelines site, the sampling approach, and the process of
hold only for the main body of the report: Tables, data collection.
figures, footnotes, and references may include ab-
breviations and Arabic numerals.
CHAPTER 21 • PREPARING A RESEARCH REPORT 535
Digital Research Tools Preliminary Pages
for the 21st Century:
The preliminary pages contain the title page,
MANAGING CITATIONS acknowledgments page, table of contents, list of
tables and figures, and abstract.
The Internet offers a variety of reference manage-
ment software choices depending on the needs, The title should communicate what the study
operating system, and budget available to you. is about. During the review of the literature for
Here we will discuss three of the most com- a study, researchers often make initial decisions
monly used and accessible citation management about the relevance of a source based on its title.
software packages available to you. A well-constructed title makes it fairly easy for
the reader to determine the nature of the topic; a
RefWorks vaguely worded one confuses the reader, who then
Many universities have adopted RefWorks as an must search through the body of the report to get
online citation management tool that they make more information. After you write your title, apply
available to their students at no cost (otherwise the communication test: Would you know what the
you will need to purchase a subscription). Ref- study was about if you read the title in an index?
Works is a commercial citation manager that pro- Ask friends or colleagues to describe what they
vides users with the ability to manage and store understand from your title.
references online at a personal database that can
be accessed and updated from any computer. It Most theses and dissertations also include an
also allows users to link to electronic editions of acknowledgments page. This page permits the
journals to which universities subscribe (perhaps writer to express appreciation to persons who have
as part of a consortium agreement) and to easily contributed significantly to the research. Notice the
capture and format bibliographic information. word significantly. You cannot (and should not!)
mention everyone who had anything to do with the
Zotero study or the report. It is acceptable to thank your
Zotero is a free plug-in for Firefox that allows users advisor for her or his guidance and assistance; it is
to instantly pull bibliographic information from web- not acceptable to thank your third-grade teacher for
sites into Zotero. For example, if you are browsing giving you confidence in your ability (remember
Amazon.com and find a book you want to add to the Academy Awards!).
your reference list, you simply click a button in your
Firefox browser window and whatever bibliographic The table of contents is an outline of your re-
information is available from the site is instantly port that indicates the page on which each major
downloaded to your personal Zotero account. You section (or chapter) and subsection begins. A list of
can later return to your account and quickly gener- tables and figures, presented on a separate page,
ate citations and references in whatever format you gives the number and title of each table and figure
choose. Similarly, if your source is an online journal and the page on which it can be found.
article, Zotero can store a copy of the source for you.
Did we mention that this is very . . . free?! Many colleges and universities require an ab-
stract, and others require a summary, but the cur-
EndNote rent trend is in favor of abstracts. The content of
EndNote is a commercial (read as pay-to-use) abstracts and summaries is identical, but the posi-
reference management software package that al- tion differs: An abstract precedes the main body of
lows users to manage bibliographies and refer- the report, and a summary follows the discussion
ences while writing your research report. And, or conclusion. Abstracts often must be limited to
EndNote X4 now meets the complete APA 6th a specific number of words, usually between 100
edition style manual requirements as well as of- and 500. Many institutions require abstracts to be
fering quick links to create footnotes, all while no more than 350 words, which is the maximum
creating lists of references in Word documents. allowed by Dissertation Abstracts International, a
repository of dissertation abstracts. The APA sets
a limit of 120 words for publication in its journals.
Because the abstract of a report is often the only
part that is read, it should briefly describe the most
536 CHAPTER 21 • PREPARING A RESEARCH REPORT
FIGURE 21.2 • Common components of a research report submitted for a
degree requirement
Title Page PRELIMINARY PAGES
Acknowledgments Page List of Tables and Figures
Table of Contents Abstract
Introduction MAIN BODY OF THE REPORT
Statement of the Problem Instruments
Review of Related Literature Design
Statement of the Hypothesis Procedure
Significance of the Study Results
Method Discussion (Conclusions and Recommendations)
Participants References (Bibliography)
APPENDIXES
Source: Format information from Publication Manual of the American Psychological Association
(5th ed., pp. 283–320), by the American Psychological Associations, Washington, DC: Author.
important aspects of the study, including the topic, The introduction begins with a description
the type of participants and instruments, the design, of the research problem or topic. A well-written
the data collection procedures, and the major results statement of a problem or topic generally indicates
and conclusions. For example, a 100-word abstract the variables examined in the study, including defi-
for a study investigating the effect of a writing- nitions of key terms that do not have commonly
oriented curriculum on the reading comprehension understood meanings. The statement of the prob-
of fourth-grade students may read as follows: lem or topic should be accompanied by a presenta-
tion of its background, including a justification for
The purpose of this study was to determine the the study in terms of its significance; that is, why
effect of a curriculum that emphasized writing should anyone care about this study?
on the reading comprehension of
fourth-grade students who were reading at The review of related literature indicates what
least one level below grade level, using a is known about the problem or topic. Its function
posttest-only control group design. After is to educate the reader about the area under study.
8 months, students (n ϭ 20) who participated The review of related literature is not a series of
in a curriculum that emphasized writing abstracts or annotations but rather a summary and
achieved significantly higher scores on the analysis of the relations and differences among rel-
reading comprehension subtest of the evant studies and reports. The review should flow
Stanford Achievement Test, Primary Level 3 in such a way that the least related references are
(grades 3.5 2 4.9) than students (n ϭ 20) who discussed first and the most related references are
did not [t (38) ϭ 4.83, p Ͻ .05]. The curriculum discussed last. The review should conclude with a
emphasizing writing was more effective in brief summary of the literature and its implications.
promoting reading comprehension.
In the report of a quantitative study, the hypoth-
The Main Body esis is presented following the literature review.
A good hypothesis clearly states the expected rela-
The body of a report contains an introduction to the tion or difference among the studied variables and
topic that also includes the review of the literature defines those variables in operational, measurable
and the hypotheses (if any); a method or procedure terms. All hypotheses logically follow the review
section describing the participants, the instruments, of related literature and are based on the impli-
and the procedures; a section presenting the re- cations of previous research. A well-developed
sults; and a discussion of the findings. hypothesis is testable—that is, it can be supported
or disconfirmed. In a report of a qualitative study,
CHAPTER 21 • PREPARING A RESEARCH REPORT 537
the researcher is unlikely to state hypotheses as observed or interviewed should be described. In
focused as those of a quantitative researcher but many cases, qualitative researchers will produce
may express some hunches about what the study more complex and detailed procedural descriptions
may show (i.e., guiding hypotheses). than quantitative researchers will.
The method section of a research report in- The results section describes the statistical
cludes a description of participants, instruments, techniques or the inferential interpretations that
design, procedure, assumptions, and limitations. were applied to the data and the results of these
For a qualitative study, this section may also include analyses. For each hypothesis, the statistical test
a detailed description of the site and the nature of significance selected and applied to the data
and length of interactions with the participants. is named, followed by the accompanying statis-
The description of participants includes informa- tics indicating whether the hypothesis was sup-
tion about how they were selected and, mainly ported or not supported. Tables and figures are
for quantitative researchers, the population they used to present findings in summary or graphic
represent. A description of the sample should indi- form and should add clarity to the presentation.
cate its size and major characteristics of members Tables present numerical data in rows and col-
such as age, grade level, ability level, and socio- umns and usually include descriptive statistics,
economic status. A good description of the sample such as means and standard deviations, and the
enables readers of the report to determine how results of tests of significance, such as t tests and
similar study participants are to other populations F ratios. Good tables and figures are uncluttered
of concern to the readers. and self-explanatory; it is better to use two tables
(or figures) than one that is crowded. They should
Data collection instruments or other materi- stand alone, that is, be interpretable without the
als used in the study should be described fully. aid of related textual material. Tables and figures
The description should indicate the purpose of the follow their related textual discussion and are re-
instrument, its application, and the validity and re- ferred to by number, not by name or location. In
liability of any instrument. If a test has been devel- other words, the text should say, “see Table 1,” not
oped by the researcher, its description needs to be “see the table with the means” or “see the table on
more detailed and should also state the manner in the next page.” Examine the variety of tables and
which it was developed, any pilot test procedures figures throughout this text to get a perspective on
and results, revisions, steps involved in scoring, how data can be presented.
and guidelines for interpretation. A copy of the
instrument, accompanying scoring keys, and other Qualitative research reporting tends to be based
pertinent information about a newly developed test mainly on descriptions and quotations that support
are generally placed as appendixes to the thesis or or illustrate the themes that emerged in the study.
dissertation proper. Charts and diagrams showing the relations among
identified topics, categories, and patterns are also
In an experimental study, the description of the useful in presenting the results of a study. The
design is especially important and should include a logic and description of the interpretations linked
rationale for its selection and a discussion of threats to qualitative charts and diagrams are important
to validity associated with the design, including aspects of qualitative research reporting.
how these threats may have been minimized in the
study being reported. In other types of research, All research reports have a section that dis-
the description of the design may be combined cusses and interprets the results, draws conclusions
with procedure. and states implications, and makes recommenda-
tions. Interpretation of results may be presented
The procedure section should describe the in a separate section titled “Discussion,” or it may
steps that the researcher(s) followed in conducting be included in the same section as the analysis of
the study, in chronological order, in sufficient de- results. What this section is called is unimportant;
tail to permit the study to be replicated by another what is important is how well it is constructed.
researcher. In essence, a step-by-step description Each result should be discussed in terms of its rela-
of what went on during the study should be pro- tion to the topic and its agreement or disagreement
vided. It should be clear how participants were with previous results obtained in other studies or
assigned to groups or treatments, if appropriate, by other researchers.
and the conditions under which participants were
538 CHAPTER 21 • PREPARING A RESEARCH REPORT
Two common errors in this section are to con- were taken, and only the primary source should be
fuse results and conclusions and to overgeneralize included in the references. For example, an in-text
results. A result is the outcome of a test of citation may be, “Nerdfais (cited in Snurd, 1995)
significance or a qualitative analysis; the corre- found that yellow chalk. . . .” No year would be
sponding conclusion is that the original hypoth- given for the Nerdfais study, and only the Snurd
esis or topic was or was not supported by the source would be listed in the references. For the-
data. In qualitative reports the conclusion may sis and dissertation studies, any sources consulted
simply be a summarizing description of what that were not directly cited in the main body of the
was observed. Researchers overgeneralize when report may be included in an appendix. The style
they state conclusions that are not warranted by manual will determine the form each reference
the data. For example, if a group of first graders must take.
receiving personalized instruction were found to
achieve significantly higher scores on a test of Appendixes are usually necessary in thesis and
reading comprehension than a group receiving dissertation reports to provide information and
traditional instruction, it would be an overgener- data that are pertinent to the study but are either
alization to conclude that personalized instruction too lengthy or not important enough to be included
is (always) a superior method of instruction for in the main body of the report. Appendixes com-
(all) elementary students. Similarly, if a qualita- monly contain materials especially developed for
tive study about teacher burnout consisted of four the study (e.g., tests, questionnaires, and cover let-
interviewees, it would be an overgeneralization to ters), raw data, and data analysis sheets.
infer that all teachers felt the same about burnout.
WRITING FOR JOURNAL
The discussion of a research report should also PUBLICATION
present the theoretical and practical implications of
the findings and make recommendations for future In the post-thesis or dissertation phase of our ac-
research or future action. In this portion of the re- ademic lives, we are invariably faced with the
port, the researcher is permitted more freedom in question, “How can I get the word out about my re-
expressing opinions that are not necessarily direct search?” Rarely do novice researchers move straight
outcomes of data analysis. The researcher is free from thesis or dissertation to best-selling book. An
to discuss any possible revisions or additions to interim step is more likely to be to present a paper
existing theory and to encourage studies designed at an academic conference and to seek public feed-
to test hypotheses suggested by the results. The back on the study. Sooner or later, however, it will
researcher may discuss implications of the find- be time to bite the bullet and prepare your research
ings for educational practice and suggest studies for your professional community—a daunting task
designed to replicate the study in other settings, for all academics, even the most seasoned. This
with other participants, and in other curricular task usually involves submission of the report to an
areas to increase the generalizability of the find- academic journal.
ings. The researcher may also suggest next-step
studies designed to investigate another dimension Writing for a journal is somewhat different
of the problem. For example, a study finding type than writing a thesis or dissertation. A major dif-
of feedback to be a factor in retention may suggest ference has to do with length. Journal articles
that amount of feedback may also be a factor and are typically shorter than theses and disserta-
recommend further research in that area. tions, although how much shorter depends on
the scope of the study, the kind of research that
The reference section of the report lists all the was conducted, the audience, the time allowed
sources that were cited in the report. Every source for writing, and other considerations. Histori-
cited in the paper must be included in the refer- cally, journals limited the length of submissions
ences, and every entry listed in the references must to 5,000 words. However, it became evident dur-
appear in the body of the paper; in other words, ing article selection for this textbook that most ar-
the citations in the manuscript and the sources in ticles are now longer—it was very difficult to find
the references must correspond exactly. If APA relatively short journal articles published after the
style is used, in-text citations of secondary sources year 2000. Furthermore, online journals routinely
should indicate the primary source from which they
CHAPTER 21 • PREPARING A RESEARCH REPORT 539
publish full-length theses on the Internet. A good manuscripts in a way that makes them engaging
rule of thumb is to follow the guidelines specified for you and your audience.
in the journal itself.
When you have a complete draft, you should
Clearly, then, selecting the appropriate journal take one last shot at making the text as tight as
is the first step. You should become very familiar possible. Wolcott2 provided a useful analogy to help
with the journal you select—take time to read with this process:
earlier volumes and pay attention to the kind of
studies published, the scope of the articles pub- Some of the best advice I’ve ever found for writ-
lished, and notification of upcoming volumes with ers happened to be included with the directions
a special guest editor or topic focus. It is possible for assembling a new wheelbarrow: Make sure
that a journal is seeking manuscripts in your par- all parts are properly in place before tightening.
ticular field—timing is everything! Journals usually
contain a section for potential contributors that We’ve never assembled a wheelbarrow, but if our
specify length guidelines, formatting rules, number experience with assembling a barbecue grill is any-
of references, and other information specific to that thing to go by, we can relate to the analogy. The
journal. Comply with the requirements and “let the directions for the barbecue were quite explicit:
manuscript speak for itself, as it will have to do “Ensure assembly is complete before igniting.” We
when published.”1 can apply the assembly metaphor to our writing
task: Be sure to take the time to read the narrative
Although journals often have idiosyncratic carefully, focusing on the details. Getting a manu-
rules for authors to follow, the standards for good script ready for publication is not the time to be
scientific writing apply for any journal you may in a hurry. You have endured many days, weeks,
select. In terms of content, think of a research and months of doing educational research. You are
article as telling a story, within the framework close to meeting your personal and professional
specified by your style manual. Attend to the goal and want to get the text off your desk (or hard
context of your study, and craft a narrative that drive). Now is not the time to be foolhardy—don’t
guides your audience to understand your goal, light the barbecue before all the pieces are correctly
your procedures, your findings, and the implica- positioned and tightened! Take time to do a word-
tions. Be sure to describe data collection, analysis by-word edit. Stated simply, Wolcott’s checklist of
and interpretation, and considerations such as things to watch for in final editing include unneces-
validity, reliability, and ethics. In terms of writing sary words; passive voice, especially forms of the
style, you should adopt a clear, reader-friendly verb to be, qualifiers such as rather, very, little, and
writing style. Don’t try to hide behind jargon, pretty; and overused phrases. Perhaps the underly-
and don’t make statements that you can’t sub- ing lesson here is that if we have a story to tell, and
stantiate. Let your data speak for themselves. Al- a compelling way in which to tell it, then there is a
though you may want to peruse the journals you very good chance that an editorial board will agree.
are considering for your submissions and notice For an engaging discussion about writing and pub-
the structure and writing style of the researchers lishing qualitative research, we highly recommend
whose work has been accepted and published, you consult Wolcott’s (2009) Writing Up Qualita-
don’t adopt a voice not your own—write using the tive Research now in its third edition.
same voice that you use to tell the story of your
research to your colleagues. Remember that you When the manuscript is polished and written to
want to keep your readers’ attention. If you are meet the guidelines of your chosen journal, you’re
like us, you read something and make a judgment almost ready to submit it. The final step is to write
like “Not bad,” “Engaging,” “Pretty bad,” or “I’ll a cover letter to send with your manuscript. In the
give it another few pages before I put it in the cover letter, briefly explain why your contribu-
round file.” For whatever reason, we intuitively tion is a good fit with the journal and how you
know what will keep our attention. Write your have satisfied the criteria specified in the “Notes to
Contributors” section of the journal. Good luck in
becoming a “published author”!
1 Writing Up Qualitative Research by H. F. Wolcott, 2009, Thou- 2 Wolcott, p. 93
sand Oaks, CA: Sage, p. 159.
540 CHAPTER 21 • PREPARING A RESEARCH REPORT
SUMMARY
GUIDELINES FOR WRITING 12. The table of contents is an outline of the
A RESEARCH REPORT report that indicates the page on which each
major section (or chapter) and subsection
1. All research reports contain a description of begins. A list of tables and figures is presented
the topic or problem, a review of literature, a on a separate page.
description of procedures, and a description of
results, but qualitative and quantitative studies 13. Most colleges and universities require an
address these topics in somewhat different abstract or summary of the study. The number
ways and give them different emphases. of pages for each will be specified and will
usually range from 100 to 500 words. The
2. Make writing part of your professional life and abstract should describe the most important
responsibility. aspects of the study, including the topic,
the type of participants and instruments, the
3. Begin with an outline that identifies and design, the procedures, and the major results
orders major topics and then differentiates and conclusions.
each major heading into logical subheadings.
14. The introduction is the first section of
4. Write, edit, and rewrite with an eye toward the main body of the report and includes
progress, not perfection. a well-written description of the problem, a
review of related literature, a statement of
5. Relate aspects of the study in a manner that the hypothesis, and definition of terms.
accurately reflects what you did and what you
found. 15. The method section includes a description of
participants, instruments, design, procedure,
6. Use clear, simple, straightforward language. assumptions, and limitations.
Correct spelling, grammar, and punctuation
are expected. 16. The description of participants in a
quantitative study includes a definition
7. Proofread the final report at least twice, or ask and description of the population from
a friend to review the manuscript for you. which the sample was selected and may
describe the method used in selecting the
FORMAT AND STYLE participants. The description of participants
in a qualitative study will include descrip-
8. Format refers to the general pattern of tions of the way participants were selected,
organization and arrangement of the report. why they were selected, and a detailed
Style refers to the rules of grammar, spelling, description of the context in which they
capitalization, and punctuation followed in function.
preparing the report.
17. The description of each instrument should
9. The Publication Manual of the American indicate the purpose of the instrument, its
Psychological Association, also called the APA application, and the validity and reliability
manual, specifies the style and format used by of any instrument.
most educational researchers.
18. The procedure section should describe the
FORMATTING THESES steps that the researcher(s) followed in
AND DISSERTATIONS conducting the study, in chronological order,
in sufficient detail to permit the study to be
10. The title of the report should describe the replicated by another researcher.
purpose of the study as clearly as possible.
19. The results section describes the statistical
11. The acknowledgments page allows the writer techniques or qualitative interpretations that
to express appreciation to persons who have were applied to the data and the results of
contributed significantly to the study. these analyses.
CHAPTER 21 • PREPARING A RESEARCH REPORT 541
20. Tables and figures are used to present 25. Appendixes include information and data
findings in summary or graph form and add that are pertinent to the study but are either
clarity to the presentation. Good tables and too lengthy or not important enough to be
figures are uncluttered and self-explanatory. included in the main body of the report.
21. Each research finding or result should WRITING FOR JOURNAL PUBLICATION
be discussed in terms of its relation to
the original research question and its 26. Journal articles are typically shorter than theses
agreement or disagreement with previous and dissertations, although how much shorter
results obtained in other studies. A result depends on factors such as the scope of the study
is the outcome of a test of significance or and the kind of research that was conducted.
a qualitative analysis; the corresponding
conclusion is that the original hypothesis 27. Selecting the appropriate journal is the first
or topic was or was not supported by the step. Follow the format and style required by
data. that journal.
22. Overgeneralization occurs when researchers 28. The standards for good scientific writing
state conclusions that are not warranted by apply for any journal you may select. Craft
the results. a narrative that guides your audience to
understand your goal, your procedures, your
23. The researcher should discuss the theoretical findings, and the implications. Let your data
and practical implications of the findings and speak for themselves. Take your time.
make recommendations for future research or
future action. 29. The final step is to write a cover letter to send
with your manuscript. In the cover letter,
24. The reference section of the report lists all briefly explain why your contribution is a
the sources that were cited in the report. The good fit with the journal.
required style manual will guide the format of
various types of references.
Go to the topic “Preparing a Research Report” in the MyEducationLab (www.myeducationlab.com) for your
course, where you can:
◆ Find learning outcomes.
◆ Complete Assignments and Activities that can help you more deeply understand the chapter content.
◆ Apply and practice your understanding of the core skills identified in the chapter with the Building
Research Skills exercises.
◆ Check your comprehension of the content covered in the chapter by going to the Study Plan. Here you
will be able to take a pretest, receive feedback on your answers, and then access Review, Practice, and
Enrichment activities to enhance your understanding. You can then complete a final posttest.
542 CHAPTER 21 • PREPARING A RESEARCH REPORT
PERFORMANCE CRITERIA TASK 10
Your research report should include all the com- able to build upon the information you developed
ponents listed in Figure 21.2, with the possible for your research plans as you write the report.
exceptions of an acknowledgments page and ap-
pendixes. For those of you who have conducted An example that illustrates the performance
quantitative studies, Task 10 involves combining called for by Task 10 appears on the following
the work you have done on previous tasks, writing pages (see Task 10 example). This example rep-
a discussion section, and preparing the appropriate resents the synthesis of the previously presented
preliminary pages (including an abstract) and ref- tasks related to the effects of interactive multime-
erences. In other words, you have already written dia on biology achievement. To the degree pos-
most of the final product for Task 10. Those of you sible with a student paper, this example follows
who have conducted qualitative studies should be the guidelines of the Publication Manual of the
American Psychological Association.
TASK 10 EXAMPLE
Effect of Interactive Multimedia on the Achievement
of 10th-Grade Biology Students
Sara Jane Calderin
Florida International University
Submitted in partial fulfillment of
the requirements for EDF 5481
April, 1994
543
Table of Contents i
List of Tables and Figures Page
ii
Abstract
iii
Introduction
Statement of the Problem 1
Review of Related Literature 1
Statement of the Hypothesis 1
2
Method
Participants 3
Instrument 3
Experimental Design 3
Procedure 3
3
Results
5
Discussion
6
References
7
544
List of Tables and Figures ii
Table
1. Means, Standard Deviation, and t Tests for the Experimental Page
5
and Control Groups 3
Figure
1. Experimental Design
545
iii
Abstract
The purpose of this study was to investigate the effect of interactive multimedia on the achievement of
10th-grade biology students. Using a posttest-only control group design and a t test for independent samples, it
was found that after approximately 8 months the students (n ϭ 30) who were instructed using interactive
multimedia achieved significantly higher scores on the biology test of the National Proficiency Survey Series
than did the students (n ϭ 30) whose instruction did not include interactive multimedia, t (58) ϭ 4.22, p Ͻ .05.
It was concluded that the interactive multimedia instruction was effective in raising the achievement level of the
participating students.
546
1
Introduction
One of the major concerns of educators and parents alike is the decline in student achievement. An area
of particular concern is science education, where the higher-level thinking skills and problem solving techniques
so necessary for success in our technological society need to be developed (Smith & Westhoff, 1992).
Research is constantly providing new proven methods for educators to use, and technology has developed
many kinds of tools ideally suited to the classroom. One such tool is interactive multimedia (IMM). IMM
provides teachers with an extensive amount of data in a number of different formats including text, sound, and
video. This makes it possible to appeal to all the different learning styles of the students and to offer a variety of
material for students to analyze (Howson & Davis, 1992).
When teachers use IMM, students become highly motivated, which results in improved class attendance
and more completed assignments (O’Connor, 1993). In addition, students also become actively involved in their
own learning, encouraging comprehension rather than mere memorization of facts (Kneedler, 1993; Reeves,
1992).
Statement of the Problem
The purpose of this study was to investigate the effect of IMM on the achievement of 10th-grade biology
students. IMM was defined as “a computerized database that allows users to access information in multiple
forms, including text, graphics, video and audio” (Reeves, 1992, p. 47).
Review of Related Literature
Due to modern technology, such as videotapes and videodiscs, students receive more information from
visual sources than they do from the written word (Helms & Helms, 1992), and yet in school the majority of
information is still transmitted through textbooks. While textbooks cover a wide range of topics superficially,
IMM can provide in-depth information on essential topics in a format that students find interesting (Kneedler,
1993). Smith and Westhoff (1992) note that when student interest is sparked, curiosity levels are increased and
students are motivated to ask questions. The interactive nature of multimedia allows students to seek out their
own answers, and by so doing they become owners of the concept involved. Ownership translates into
comprehension (Howson & Davis, 1992).
Many science concepts are learned through observation of experiments. By using IMM, students can
participate in a variety of experiments which are either too expensive, too lengthy, or too dangerous to carry out
in the school laboratory (Howson & Davis, 1992; Leonard, 1989; Louie, Sweat, Gresham, & Smith, 1991).
While observing experiments students can discuss what is happening and ask questions. At the touch of a button
teachers are able to replay any part of the proceedings, and they also have random access to related information
which can be used to illustrate completely the answer to the question (Howson & Davis, 1992). By answering
students’ questions in this detailed way, the content becomes more relevant to the needs of the students (Smith &
Westhoff, 1992). When knowledge is relevant students are able to use it to solve problems and in so doing
develop higher-level thinking skills (Helms & Helms, 1992; Sherwood, Kinzer, Bransford, & Franks, 1987).
547
2
A major challenge of science education is to provide students with large amounts of information that will
encourage them to be analytical (Howson & Davis, 1992; Sherwood et al., 1987). IMM offers electronic access
to extensive information allowing students to organize, evaluate, and use it in the solution of problems (Smith &
Wilson, 1993). When information is introduced as an aid to problem solving it becomes a tool with which to
solve other problems, rather than a series of solitary, disconnected facts (Sherwood et al., 1987).
Although critics complain that IMM is entertainment and students do not learn from it (Corcoran, 1989),
research has shown that student learning does improve when IMM is used in the classroom (Sherwood et al.,
1987; Sherwood & Others, 1990). A 1987 study by Sherwood et al., for example, showed that seventh- and
eighth-grade science students receiving instruction enhanced with IMM had better retention of that information,
and O’Connor (1993) found that the use of IMM in high school mathematics and science increased the focus on
students’ problem solving and critical thinking skills.
Statement of the Hypothesis
The quality and quantity of software available for science classes has dramatically improved during the
past decade. Although some research has been carried out on the effects of IMM on student achievement in
science, due to promising updates in the technology involved, further study is warranted. Therefore, it was
hypothesized that 10th-grade biology students whose teachers use IMM as part of their instructional technique
will exhibit significantly higher achievement than 10th-grade biology students whose teachers do not use IMM.
548
3
Method
Participants
The sample for this study was selected from the total population of 213 10th-grade students at an upper
middle class all-girls Catholic high school in Miami, Florida. The population was 90% Hispanic, mainly of
Cuban-American descent. Sixty students were randomly selected (using a table of random numbers) and
randomly assigned to two groups of 30 each.
Instrument
The biology test of the National Proficiency Survey Series (NPSS) was used as the measuring instrument.
The test was designed to measure individual student performance in biology at the high school level but the
publishers also recommended it as an evaluation of instructional programs. Content validity is good; items were
selected from a large item bank provided by classroom teachers and curriculum experts. High school
instructional materials and a national curriculum survey were extensively reviewed before objectives were
written. The test objectives and those of the biology classes in the study were highly correlated. Although the
standard error of measurement is not given for the biology test, the range of KR-20s for the entire battery is from
.82 to .91 with a median of .86. This is satisfactory since the purpose of the test was to evaluate instructional
programs, not to make decisions concerning individuals. Catholic school students were included in the battery
norming procedures, which were carried out in April and May of 1988 using 22,616 students in grades 9–12
from 45 high schools in 20 states.
Experimental Design
The design used in this study was the posttest-only control group design (see Figure 1). This design was
selected because it provides control for most sources of invalidity and random assignment to groups was
possible. A pretest was not necessary since the final science grades from June 1993 were available to check
initial group equivalence and to help control mortality, a potential threat to internal validity with this design.
Mortality, however, was not a problem as no students dropped from either group.
Figure 1. Experimental design.
Group Assignment n Treatment Posttest
1 Random 30 IMM instruction NPSS:Ba
2 Random 30 Traditional instruction NPSS:B
aNational Proficiency Survey Series: Biology
Procedure
Prior to the beginning of the 1993–1994 school year, before classes were scheduled, 60 of the 213 10th-
grade students were randomly selected and randomly assigned to two groups of 30 each, the average biology
549
4
class size; each group became a biology class. One of the classes was randomly chosen to receive IMM
instruction. The same teacher taught both classes.
The study was designed to last eight months beginning on the first day of class. The control group was
taught using traditional methods of lecturing and open class discussions. The students worked in pairs for
laboratory investigations, which included the use of microscopes. The teacher’s role was one of information
disseminator.
The experimental classroom had 15 workstations for student use, each one consisting of a laserdisc
player, a video recorder, a 27-inch monitor, and a Macintosh computer with a 40 MB hard drive, 128 MB RAM,
and a CD-ROM drive. The teacher’s workstation incorporated a Macintosh computer with CD-ROM drive, a
videodisc player, and a 27-inch monitor. The workstations were networked to the school library so students had
access to online services such as Prodigy and Infotrac as well as to the card catalogue. Two laser printers were
available through the network for the students’ use.
In the experimental class the teacher used a videodisc correlated to the textbook. When barcodes provided
in the text were scanned, a section of the videodisc was activated and appeared on the monitor. The section
might be a motion picture demonstrating a process or a still picture offering more detail than the text. The role of
the teacher in the experimental group was that of facilitator and guide. After the teacher had introduced a new
topic, the students worked in pairs at the workstations investigating topics connected to the main idea presented
in the lesson. Videodiscs, CD-ROMs, and online services were all available as sources of information. The
students used HyperStudio to prepare multimedia reports, which they presented to the class.
Throughout the study the same subject matter was covered and the two classes used the same text.
Although the students of the experimental group paired up at the workstations, the other group worked in pairs
during lab time, thus equalizing any effect from cooperative learning. The classes could not meet at the same
time as they were taught by the same teacher, so they met during second and third periods. First period was not
chosen as the school sometimes has a special schedule which interferes with first period. Both classes had the
same homework reading assignments, which were reviewed in class the following school day. Academic
objectives were the same for each class and all tests measuring achievement were identical.
During the first week of May, the biology test of the NPSS was administered to both classes to compare
their achievement in biology.
550
5
Results
Prior to the beginning of the study, after the 60 students were randomly selected and assigned to
experimental and control groups, final science grades from the previous school year were obtained from school
records in order to check initial group equivalence. Examination of the means and a t test for independent
samples (␣ ϭ .05) indicated essentially no difference between the groups (see Table 1). A t test for independent
samples was used because the groups were randomly formed and the data were interval.
Table 1
Means, Standard Deviation, and t Tests for the Experimental and Control Groups
Group
Score IMM instructiona Traditional instructiona t
Prior 87.47 87.63 Ϫ 0.08*
Grades 8.19 8.05 4.22**
M 32.27 26.70
SD 4.45 5.69
Posttest
NPSS:B
M
SD
Note. Maximum score for prior grades ϭ 100. Maximum score for posttest ϭ 40.
an ϭ 30.
*p > .05. **p < .05.
At the completion of the eight-month study, during the first week in May, scores on the NPSS:B were
compared, also using a t test for independent samples. As Table 1 indicates, scores of the experimental and
control groups were significantly different. In fact, the experimental group scored approximately one standard
deviation higher than the control group (ES ϭ .98). Therefore, the original hypothesis that “10th-grade biology
students whose teachers use IMM as part of their instructional technique will exhibit significantly higher
achievement than 10th-grade biology students whose teachers do not use IMM” was supported.
551
6
Discussion
The results of this study support the original hypothesis: 10th-grade biology students whose teachers used
IMM as part of their instructional technique did exhibit significantly higher achievement than 10th-grade
biology students whose teachers did not use IMM. The IMM students’ scores were 5.57 points (13.93%) higher
than those of the other group. Also, it was informally observed that the IMM instructed students were eager to
discover information on their own and to carry on the learning process outside scheduled class hours.
Results cannot be generalized to all classrooms because the study took place in an all-girls Catholic high
school with the majority of the students having an Hispanic background. However, the results were consistent
with research on IMM in general, and in particular with the findings of Sherwood et al. (1987) and O’Connor
(1993) concerning the improvement of student achievement.
IMM appears to be a viable educational tool with applications in a variety of subject areas and with both
cognitive and psychological benefits for students. While further research is needed, especially using other
software and in other subject areas, the suggested benefits to students’ learning offered by IMM warrant that
teachers should be cognizant of this instructional method. In this technological age it is important that education
take advantage of available tools which increase student motivation and improve academic achievement.
552
7
References
Corcoran, E. (1989, July). Show and tell: Hypermedia turns information into a multisensory event. Scientific
American, 261, 72, 74.
Helms, C. W., & Helms, D. R. (1992, June). Multimedia in education (Report No. IR-016-090). Proceedings of
the 25th Summer Conference of the Association of Small Computer Users in Education. North Myrtle
Beach, SC. (ERIC Document Reproduction Service No. ED 357 732)
Howson, B. A., & Davis, H. (1992). Enhancing comprehension with videodiscs. Media and Methods, 28(3),
12–14.
Kneedler, P. E. (1993). California adopts multimedia science program. Technological Horizons in Education
Journal, 20(7), 73–76.
Lehmann, I. J. (1990). Review of National Proficiency Survey Series. In J. J. Kramer & J. C. Conoley (Eds.),
The eleventh mental measurements yearbook (pp. 595–599). Lincoln: University of Nebraska, Buros
Institute of Mental Measurement.
Leonard, W. H. (1989). A comparison of student reaction to biology instruction by interactive videodisc or
conventional laboratory. Journal of Research in Science Teaching, 26, 95–104.
Louie, R., Sweat, S., Gresham, R., & Smith, L. (1991). Interactive video: Disseminating vital science and math
information. Media and Methods, 27(5), 22–23.
O’Connor, J. E. (1993, April). Evaluating the effects of collaborative efforts to improve mathematics and science
curricula (Report No. TM-019-862). Paper presented at the Annual Meeting of the American Educational
Research Association, Atlanta, GA. (ERIC Document Reproduction Service No. ED 357 083)
Reeves, T. C. (1992). Evaluating interactive multimedia. Educational Technology, 32(5), 47–52.
Sherwood, R. D., Kinzer, C. K., Bransford, J. D., & Franks, J. J. (1987). Some benefits of creating macro-
contexts for science instruction: Initial findings. Journal of Research in Science Teaching, 24, 417–435.
Sherwood, R. D., & Others. (1990, April). An evaluative study of level one videodisc based chemistry program
(Report No. SE-051-513). Paper presented at a Poster Session at the 63rd Annual Meeting of the National
Association for Research in Science Teaching, Atlanta, GA. (ERIC Document Reproduction Service No.
ED 320 772)
Smith, E. E., & Westhoff, G. M. (1992). The Taliesin project: Multidisciplinary education and multimedia.
Educational Technology, 32, 15–23.
Smith, M. K., & Wilson, C. (1993, March). Integration of student learning strategies via technology (Report No.
IR-016-035). Proceedings of the Fourth Annual Conference of Technology and Teacher Education. San
Diego, CA. (ERIC Document Reproduction Service No. ED 355 937)
553
CHAPTER TWENTY-TWO
Goldfinger, 1964
“A researcher critically evaluates each
reference and does not consider poorly executed
research.” (p. 555)
Evaluating
a Research Report
LEARNING OUTCOMES
After reading Chapter 22, you should be able to do the following:
1. Evaluate each of the major sections and subsections of a research report.
2. For each type of research, evaluate the adequacy of a study representing that
type.
These outcomes form the basis for the following task, which requires you to evalu-
ate a research report.
TASK 11
Given a reprint of a research report and an evaluation form, evaluate the components
of the report (See Performance Criteria, p. 566).
Knowing how to conduct research and how to produce a research report are
valuable skills, but as a professional you should also know how to consume and
evaluate research. Anyone who reads a newspaper, listens to the radio, or watches
television is a consumer of research. Many people uncritically accept and act on
medical and health findings, for example, because the findings are presented by
someone in a white lab coat or because they are labeled “research.” Very few people
question the procedures utilized or the generalizability of the findings. You have a
responsibility to be informed about the latest findings in your professional area and
to be able to differentiate good from poor research when investigating a topic to
study. A researcher critically evaluates each reference and does not consider poorly
executed research.
To evaluate a research study competently, you must have knowledge of each
component of the research process. Your work in previous chapters has given you
that knowledge. In this chapter, then, we discuss the criteria on which to evaluate
a research report.
GENERAL EVALUATION CRITERIA
Many research studies have flaws of various kinds. Just because a study is
published does not necessarily mean that it is a good study or that it is reported
adequately. The most common flaw is a failure to collect or report validity and
reliability information about data-gathering procedures and instruments such
as tests, observations, questionnaires, and interviews. Other common flaws in
555
556 CHAPTER 22 • EVALUATING A RESEARCH REPORT
a study itself include weaknesses in the research Despite the lack of complete agreement in some
design and inappropriate or biased selection of areas, evaluation of a research report is a worthwhile
participants; flaws in the report include failure and important activity. Major problems and short-
to state limitations in the research and a general comings are usually readily identifiable, and you can
lack of description about the study. Watching formulate an overall impression of the quality of the
for these problems is part of being a competent study. In the sections that follow, we list for your
consumer of research reports; the problems also consideration evaluative questions about a number
highlight common pitfalls to avoid in your own of research strategies and areas. This list is by no
research. means exhaustive, and as you read it, you may very
well think of additional questions to ask. You may
At your current level of expertise, you may also note that not every criterion applies equally to
not be able to evaluate every component of every both quantitative and qualitative research studies.
study. For example, you may not be able to deter-
mine whether the appropriate degrees of freedom Introduction
were used in the calculation of an analysis of co-
variance. However, you should be able to detect a Problem
number of basic errors or weaknesses in research
studies. You should, for example, be able to iden- ■ Is there a statement of the problem? Does the
tify the threats to validity associated with a study problem indicate a particular focus of study?
that has a one-group pretest-posttest design. You
should also be able to detect obvious indications of ■ Is the problem researchable? That is, can it be
experimenter bias that may have affected qualita- investigated by collecting and analyzing data?
tive or quantitative research results. For example,
a statement in a research report that “the purpose ■ Is background information on the problem
of this study was to prove . . .” should alert you to presented?
a probable bias.
■ Is the educational significance of the problem
As you read a research report, either as a con- discussed?
sumer of research keeping up with the latest find-
ings in your professional area or as a producer of ■ Does the problem statement indicate the
research reviewing literature related to a defined variables of interest and the specific relations
problem, you should ask and answer a number of among the variables that were investigated?
questions about the adequacy of a study and its
components. The answers to some of these ques- ■ When necessary, are variables directly or
tions are more critical than the answers to others. operationally defined?
An inadequate title is not a critical flaw; an inad-
equate research plan is. Some questions are dif- ■ Did the researcher have the knowledge and
ficult to answer if the study is not directly in your skill to carry out the research?
area of expertise. If your area of specialization is
reading, for example, you are probably not in a Review of Related Literature
position to judge the adequacy of a review of lit-
erature related to anxiety effects on learning. And, ■ Is the review comprehensive?
admittedly, the answers to some questions are ■ Is the review well organized? Does it logically
more subjective than objective. Whether a study
was well designed is pretty clear and objective; flow in such a way that the references least
most quantitative researchers would agree that the related to the problem are discussed first and
randomized posttest-only control group design is those most related are discussed last? Does
a good design. On the other hand, the answer to it educate the reader about the problem or
whether the most appropriate design was used, topic?
given the problem under study, often involves a de- ■ Is the review more than a series of abstracts
gree of subjective judgment. For example, the need or annotations? That is, have the references
for a pretest may be a debatable point; it depends on been analyzed and critiqued and the results of
the study and its design. various studies compared and contrasted?
■ Are all cited references relevant to the problem
under investigation? Is the relevance of each
reference explained?
■ Does the review conclude with a summary
and interpretation of the literature and its
implications for the problem under study?
CHAPTER 22 • EVALUATING A RESEARCH REPORT 557
■ Do the implications form an empirical or ■ If appropriate, are subtest reliabilities given?
theoretical rationale for the hypotheses that ■ Is evidence presented to indicate that the
follow?
instruments are appropriate for the intended
■ Are most of the sources primary (i.e., are there sample? For example, is the reading level of an
only a few or no secondary sources)? instrument suitable for sample participants?
■ If an instrument was developed specifically for
■ Are references cited completely and accurately? the study, are the procedures involved in its
development and validation described?
Hypotheses ■ If an instrument was developed specifically
for the study, are administration, scoring or
■ Are specific research questions listed or specific tabulating, and interpretation procedures fully
hypotheses stated? described?
■ Does the researcher have the needed skills
■ Is each hypothesis testable? or experience to construct or administer an
■ Does each hypothesis state an expected relation instrument?
or difference? Design and Procedure
■ If necessary, are variables directly or
■ Are the design and procedures appropriate for
operationally defined? examining the research question or testing the
hypotheses of the study?
Method
■ Are the procedures described in sufficient detail
Participants to permit replication by another researcher?
■ Are the size and major characteristics of the ■ Do procedures logically relate to one another?
population described? ■ Were instruments and procedures applied
■ If a sample was selected, is the method of correctly?
selecting the sample clearly described? ■ If a pilot study was conducted, are its execution
■ Does the method of sample selection suggest and results described? Is the effect on the
any limitations or biases in the sample? For subsequent study explained?
example, was stratified sampling used to obtain ■ Are control procedures described?
sample subgroups? ■ Does the researcher discuss or account for any
potentially confounding variable that he or she
■ Are the size and major characteristics of the was unable to control?
sample described?
Results
■ If the study is quantitative, does the sample size
meet the suggested guidelines for the minimum ■ Are appropriate descriptive statistics presented?
sample size appropriate for the method of ■ Are the tests of significance appropriate, given
research represented?
the hypotheses and design of the study?
Instruments ■ If parametric tests were used, is there evidence
■ Do instruments and their administration meet that the researcher avoided violating the
guidelines for protecting human subjects? Were required assumptions for parametric tests?
needed permissions obtained? ■ Was the probability level at which the tests
of significance were evaluated specified in
■ Are the instruments appropriate for measuring advance of the data analyses? Was every
the intended variables? hypothesis tested?
■ Are the tests of significance interpreted using
■ Was the correct type of instrument used for the appropriate degrees of freedom?
data collection (e.g., was a norm-referenced ■ Was the inductive logic used to produce results
instrument used when a criterion-referenced in a qualitative study made explicit?
one was more suitable)?
■ Is the rationale given for the selection of the
instruments (or measurements) used?
■ Are the purpose, content, validity, and
reliability of each instrument described?
558 CHAPTER 22 • EVALUATING A RESEARCH REPORT
■ Are the results clearly described? single-subject), some to mixed methods research,
■ Are the tables and figures (if any) well and some to action research. Both quantitative
and qualitative criteria may be applied by varying
organized and easy to understand? degrees to mixed methods research depending on
■ Are the data in each table and figure described the emphasis placed on quantitative and qualitative
research methods. The following sections list some
in the text? of those questions for each study type.
Discussion (Conclusions Survey Research
and Recommendations)
■ Are questionnaire validation procedures
■ Is each result discussed in terms of the original described?
hypothesis or topic to which it relates?
■ Was the questionnaire pilot tested?
■ Is each result discussed in terms of its ■ Are pilot study procedures and results
agreement or disagreement with previous
results obtained by other researchers in other described?
studies? ■ Are directions to questionnaire respondents
■ Are generalizations consistent with the results? clear?
■ Are theoretical and practical implications of the ■ Does each item in the questionnaire relate to
findings discussed? an objective of the study?
■ Are the possible effects of uncontrolled ■ Does each questionnaire item deal with a single
variables on the results discussed? concept?
■ Are recommendations for future action made? ■ When necessary, is a point of reference given
■ Are the suggestions for future action based
for questionnaire items?
on practical significance or on statistical ■ Are leading questions avoided in the
significance only (i.e., has the author avoided
confusing practical and statistical significance)? questionnaire?
■ Are there sufficient alternatives for each
Abstract or Summary
questionnaire item?
(Note: It is easier to review the abstract after you ■ Does the cover letter explain the purpose
have read the rest of the report.)
and importance of the study, and does it give
■ Is the problem stated? the potential respondent a good reason for
■ Are the number and type of participants and cooperating?
■ If appropriate, is confidentiality or anonymity
instruments described? of responses assured in the cover letter?
■ Is the design identified? ■ What is the percentage of returns, and how
■ Are procedures described? does it affect the study results?
■ Are the major results and conclusions stated? ■ Are follow-up activities to increase returns
described?
TYPE-SPECIFIC EVALUATION ■ If the response rate was low, was any attempt
CRITERIA made to determine any major differences
between respondents and nonrespondents?
In addition to general criteria that can be applied ■ Are data analyzed in groups or clusters rather
to almost any study, there are additional questions than in a series of many single-variable
you should ask depending on the type of research analyses?
represented by the study. In other words, some con-
cerns are specific to qualitatively oriented research Correlational Research
(e.g., narrative, ethnographic, case study), some to
quantitatively oriented research (e.g., survey, cor- Relationship Studies
relational, causal–comparative, experimental, and
■ Were variables carefully selected (i.e., was a
shotgun approach avoided)?
■ Is the rationale for variable selection described?
CHAPTER 22 • EVALUATING A RESEARCH REPORT 559
■ Are conclusions and recommendations based ■ Was condition or phase length sufficient
on values of correlation coefficients corrected to represent the behavior within the
for attenuation or restriction in range? phase?
■ Do the conclusions avoid suggesting causal ■ Is the design appropriate to the question under
relations among the variables investigated? study?
Prediction Studies ■ If a multiple baseline design was used, were
conditions met to move across baselines?
■ Is a rationale given for selection of predictor
variables? ■ If a withdrawal design was used, are limitations
to this design addressed?
■ Is the criterion variable well defined?
■ Was the resulting prediction equation validated ■ Did the researcher manipulate only one
variable at a time?
with at least one other group?
■ Is the study replicable?
Causal–Comparative Research
Qualitative Research (in General)
■ Are the characteristics or experiences that
differentiate the groups (i.e., the grouping ■ Does the researcher give a general sense of the
variable) clearly defined or described? focus of study?
■ Are critical extraneous variables identified? ■ Does the researcher state a guiding hypothesis
■ Were any control procedures applied to equate for the investigation?
the groups on extraneous variables? ■ Is the application of the qualitative method
■ Are causal relations discussed with due caution? described in detail?
■ Are plausible alternative hypotheses discussed?
■ Is the context of the qualitative study described
Experimental Research in detail?
■ Was an appropriate experimental design ■ Is the purposive sampling procedure described
selected? and related to the study focus?
■ Is a rationale for design selection given? ■ Is each data collection strategy described?
■ Are threats to validity associated with the ■ Is the researcher’s role stated (e.g.,
design identified and discussed? nonparticipant observer, participant observer,
■ Is the method of group formation described? interviewer, etc.)?
■ Was the experimental group formed in the ■ Are the research site and the researcher’s entry
into it described?
same way as the control group? ■ Were the data collection strategies used
■ Were groups randomly formed and the use of appropriately, given the purpose of the
study?
existing groups avoided? ■ Were strategies used to strengthen the
■ Were treatments randomly assigned to groups? validity and reliability of the data (e.g.,
■ Were critical extraneous variables identified? triangulation)?
■ Were any control procedures applied to equate ■ Is there a description of how any unexpected
ethical issues were handled?
groups on extraneous variables? ■ Are strategies used to minimize observer bias
■ Were possible reactive arrangements (e.g., the and observer effect described?
■ Are the researcher’s reactions and notes
Hawthorne effect) controlled for? differentiated from descriptive field notes?
■ Are the results generalized to the appropriate ■ Are data coding strategies described and
examples of coded data given?
group? ■ Is the inductive logic applied to the data to
produce results stated in detail?
Single-Subject Research ■ Are conclusions supported by data (e.g., are
direct quotations from participants used to
■ Are the data time constrained? illustrate points)?
■ Was a baseline established before moving into
the intervention phase?
560 CHAPTER 22 • EVALUATING A RESEARCH REPORT
Evaluating Validity and Reliability and/or acculturation, and pertinent social and
in Qualitative Studies1 cultural history?
■ Are researcher interaction effects addressed by
Threats to Internal Validity fully documenting the researcher’s activities in
the setting?
■ Did the researcher effectively deal with ■ Were all observations and interviews conducted
problems of history and maturation by in a variety of fully described settings and with
documenting historical changes over time? multiple trained observers?
■ Did the researcher effectively deal with Reliability
problems of mortality by using a sample large
enough to minimize the effects of attrition? ■ Is the researcher’s relationship with the group
and setting fully described?
■ Was the researcher in the field long enough to
minimize observer effects? ■ Is all field documentation comprehensive, fully
cross-referenced and annotated, and rigorously
■ Did the researcher take the time to become detailed?
familiar and comfortable with participants?
■ Were observations and interviews documented
■ Were interview questions pilot tested? using multiple means (e.g., written notes and
■ Were efforts made to ensure intraobserver recordings)?
agreement by training interview teams in ■ Was the interviewer’s training documented, and
coding procedures? is it described?
■ Were efforts made to cross-check results by
conducting interviews with multiple groups? ■ Was the construction, planning, and testing
■ Did the researcher interview key informants to of all instruments documented, and are they
verify field observations? described?
■ Were participants demographically screened
to ensure that they were representative of the ■ Are key informants fully described, and is
larger population? information on groups they represent and their
■ Were data collected using different media community status included?
(e.g., audiotape, videotape, etc.) to facilitate
cross-validation? ■ Are sampling techniques fully documented and
■ Were participants allowed to evaluate research sufficient for the study?
results before publication?
■ Are sufficient data presented to support Narrative Research
findings and conclusions?
■ Were variables repeatedly tested to validate ■ Does the researcher provide a rationale for the
results? use of narrative research to study the chosen
phenomenon?
Threats to External Validity
■ Is there a rationale for the choice of individual
■ Were constructs defined in a way that has to study the chosen phenomenon?
meaning outside the setting of the study?
■ Does the researcher describe data collection
■ Were both new and adapted instruments pilot methods and give particular attention to
tested to ensure that they were appropriate for interviewing?
the study?
■ Does the researcher describe appropriate
■ Does the researcher fully describe participants’ strategies for analysis and interpretation (e.g.,
relevant characteristics, such as socioeconomic restorying)?
structure, gender makeup, level of urbanization
Ethnographic Research
1 The questions in this section were adapted from Ethnographer’s
Toolkit: Vol. 2. Essential Ethnographic Methods: Observations, ■ Does the written account (i.e., the ethnography)
Interviews, and Questionnaires (pp. 278–289), by S. L. Schensul, capture the social, cultural, and economic
J. J. Schensul, and M. D. LeCompte, 1999, Lanham, MD: AltaMira/ themes that emerged from the study?
Rowman & Littlefield.
■ Did the researcher spend a full cycle in the
field studying the phenomenon?
CHAPTER 22 • EVALUATING A RESEARCH REPORT 561
Case Study Research ■ Are qualitative and quantitative data collection
techniques clearly identified?
■ Was the phenomenon under investigation
appropriate for investigation using a case study ■ Are the data analysis techniques appropriate
research method? for the type of mixed methods design?
■ Is there a rationale for the selection of the case ■ Was the study feasible given the amount of
(i.e., unit of analysis)? data to be collected and concomitant issues
of resources, time, and expertise?
■ Does the researcher provide a clear description
of the case? Action Research
■ Was an appropriate analysis of the case, or ■ Does the area of focus involve teaching and
cross-site analysis, conducted? learning in the researcher’s own practice?
■ Is there a clear link between the data presented ■ Was the area of focus within the researcher’s
in the case study and the themes that are locus of control?
reported?
■ Is the area of focus something the researcher
Mixed Methods Research was passionate about?
■ Did the study use at least one quantitative and ■ Is the area of focus something the researcher
at least one qualitative data research method? wanted to change or improve upon?
■ Did the study investigate both quantitative and ■ Does the researcher state questions that were
qualitative research questions? answerable given the researcher’s expertise,
time, and resources?
■ Is a rationale for using a mixed methods
research design provided? ■ Does the researcher provide an action plan
detailing the effect of the research findings
■ Is the type of mixed methods research design on practice?
stated?
■ Is the priority given to quantitative and
qualitative data collection and the sequence of
their use described?
562 CHAPTER 22 • EVALUATING A RESEARCH REPORT
SUMMARY
GENERAL EVALUATION CRITERIA Survey Research
16. See page 558.
1. You should be able to detect a number of
basic errors or weaknesses in a research study. Correlational Research
17. Relationship studies: See page 558.
2. You should be able to detect obvious 18. Prediction studies: See page 559.
indications of experimenter bias that may
have affected the results. Causal–Comparative Research
19. See page 559.
3. As you read a research report, you should
ask questions concerning the execution of the Experimental Research
study. 20. See page 559.
4. The answers to some questions are more Single-Subject Research
critical than the answers to others. 21. See page 559.
5. Major problems and shortcomings are usually Qualitative Research (In General)
readily identifiable, and you can formulate an 22. See page 559.
overall impression of the quality of the study.
Evaluating Validity and Reliability
Introduction in Qualitative Studies
6. Problem: See page 556. 23. See page 560.
7. Review of related literature: See page 556.
8. Hypotheses: See page 557. Narrative Research
24. See page 560.
Method
9. Participants: See page 557. Ethnographic Research
10. Instruments: See page 557. 25. See page 560.
11. Design and procedure: See page 557.
Case Study Research
Results 26. See page 561.
12. See page 557.
Mixed Methods Research
Discussion (Conclusions and Recommendations) 27. See page 561.
13. See page 558.
Action Research
Abstract or Summary 28. See page 561.
14. See page 558.
TYPE-SPECIFIC EVALUATION CRITERIA
15. In addition to general criteria that can be
applied to almost any study, additional
questions should be asked depending on the
type of research represented by the study.
CHAPTER 22 • EVALUATING A RESEARCH REPORT 563
Go to the topic “Evaluating a Research Report” in the MyEducationLab (www.myeducationlab.com) for your
course, where you can:
◆ Find learning outcomes.
◆ Complete Assignments and Activities that can help you more deeply understand the chapter content.
◆ Apply and practice your understanding of the core skills identified in the chapter with the Building
Research Skills exercises.
◆ Check your comprehension of the content covered in the chapter by going to the Study Plan. Here you
will be able to take a pretest, receive feedback on your answers, and then access Review, Practice, and
Enrichment activities to enhance your understanding. You can then complete a final posttest.
564 CHAPTER 22 • EVALUATING A RESEARCH REPORT
PERFORMANCE CRITERIA TASK 11
The research report to be evaluated appears on the In addition (and where possible), indicate where
following pages (see Task 11 Example). Immedi- you found the answer to the question in the report.
ately following is the form you should use in evalu- For example, if asked if a hypothesis is stated in the
ating the report (see Self-Test for Task 11). Answer report, and the answer is yes, you may write “para-
each question by writing one of the following on graph 7, sentences 6 and 7” as the location of the
the line in the “Code” column: information. Write the information in the margin
next to the question.
Y ϭ Yes
Suggested responses for the Self-Test for Task 11
N ϭ No appear in Appendix C.
NA ϭ Question not applicable (e.g., a pilot
study was not done)
?/X ϭ Cannot tell from information given or,
given your current level of expertise,
you are not in a position to make a
judgment
TASK 11 Example
Gender and Race as Variables in Psychosocial Adjustment
to Middle and High School
PATRICK AKOS
JOHN P. GALASSI
The University of North Carolina at Chapel Hill
ABSTRACT School transition research indicates that during the transition to middle school (Blyth et al., 1983;
negative outcomes (e.g., decreases in self-esteem and Eccles et al., 1993). Those findings may be due to the
academic motivation) occur for a number of students in fact that girls experience more peer upheaval or that they
transition. Although data are not consistent, gender and experience more distress than do boys because of the
race tend to play a role in school transition outcomes. greater salience of their peer networks. The combination
The authors investigated gender and race as variables of physical and social transitions that girls experience
in 6th- and 9th-grade students’ psychosocial adjustment from elementary to middle school also may contribute to
(e.g., perceptions of difficulty of transition and connect- their distress. Because girls typically mature earlier than
edness to school) following a recent school transition boys (Eccles et al.), they often make a distinctive physical
and in persons who they perceived as helpful in the transition (e.g., menstruation) into puberty simultane-
transition process. Results suggest differences by gender ously with a school transition. Thus, the combination
for feelings of connectedness to middle and high school of and feelings about the two transitions—physical and
following the transition. Latino students perceived the school—may heighten the negative outcomes that some
transition to middle school as significantly more difficult girls experience as they enter middle school.
than did Caucasian and African American students. Ad-
ditional findings and implications are presented. Researchers also have demonstrated that girls expe-
rience more depression than do boys over the transi-
Key words: gender and race, psychosocial adjustment, tion from elementary to middle school (Blyth et al.,
transition to middle and high school 1983; Hirsch & Rapkin, 1987). Moreover, Diemert (1992)
found that boys reported a lack of assistance with aca-
Most education researchers have suggested that school demic needs, whereas girls reported a lack of assistance
transitions play an important role in the developmental with social needs in the middle school transition. In the
trajectory of students (Eccles et al., 1993; Simmons & Blyth, high school transition, girls exhibited more concerns in
1987; Wigfield, Eccles, Mac Iver, Reuman, & Midgley, 1991). general and more intense social and academic concerns,
The individual or personal transformations that students whereas boys were concerned particularly about hav-
undergo during puberty and school changes are exten- ing longer class periods, participating in sports, and
sive and frequently disruptive. Researchers have identified violence from gangs (Maute, 1991). Although most of
declines in academic performance (Blyth, Simmons, & the transition research has suggested differential out-
Carlton-Ford, 1983), academic motivation (Eccles et al.), comes based on gender, some researchers failed to rep-
self-esteem (Simmons & Blyth), extracurricular participa- licate these differences (Seidman et al., 1994; Wampler,
tion (Seidman, Allen, Aber, Mitchell, & Feinman, 1994), and Munsch, & Adams, 2002).
perceived support from school staff (Seidman et al.) as well
as increases in daily hassles (Seidman et al.), as some of the In addition to gender, race has been another focus of
negative effects of school transitions. transition research. Several researchers suggested that
minority status may intensify negative transition out-
However, only a modest number of researchers have comes (Cauce, Hannan, & Sargeant, 1992; Mosely & Lex,
examined the influence of demographic variables in 1990). Seidman and colleagues (1994) speculated that ur-
school transitions. Several researchers suggested that girls ban minority students often are located in overcrowded
suffer greater losses in self-esteem compared with boys classrooms in large schools that are entrenched in red
tape. They hypothesized that environmental conditions
Address correspondence to Patrick Akos, CB #3500, The Univer- of high poverty and less space intensify the contextual
sity of North Carolina at Chapel Hill, Chapel Hill, NC 27599- transition and can potentially lead to more detrimental
3500. (E-mail: [email protected]) effects such as disproportionally high rates of education
failure for urban minority youth (Seidman et al.).
Gutman and Midgley (2000) also investigated transi-
tion effects on African American students and found sig-
nificant achievement losses from elementary to middle
565
school. Simmons, Black, and Zhou (1991) also suggested these variables. Determining who different groups of
that grade declines may be more severe for African students find helpful during transitions should pro-
American students than for European American stu- vide practitioners with useful information to facilitate
dents. Simmons and colleagues discovered that African more positive school transition experiences for all
American students showed greater decreases in grade students.
point average (GPA) and more dislike for school after
the elementary to middle school transition. Maute (1991) To develop effective school transition programming,
found that in high school, Asian students and those school personnel may need to consider the influence of
defined as “Other” (not White, Black, or Latino) dem- race and gender as variables in the transition process.
onstrated more intense high school concerns than did Therefore, we explored those variables in students’ psy-
other students. Students labeled “Other” were concerned chosocial adjustment to middle and high school transi-
especially about being with children of other races and tions. Perceptions of transition difficulty and of school
being liked by others. In addition, gender and racial dif- connectedness served as gauges of adjustment to the
ferences in transition concerns varied according to the new school because adjustment has been a significant
school attended (Maute). factor in the developmental pathways of students. More-
over, the sources of support that different groups of stu-
Wampler and colleagues (2002) examined racial dif- dents find helpful in transitions have rather immediate
ferences in GPA trajectories during the transition to consequences in the way that students approach school
junior high. They discovered that grade trajectories dif- as well as in the implications for future school transition
fered for race but not for gender. Specifically, African programming by school personnel.
American students’ GPAs remained steady over the tran-
sition, Caucasian students experienced slight declines, The primary research question of the study was:
and Latino students faced steep declines with some Are race and gender significant variables in (a) overall
rebound effect near the end of the school year. student perception/evaluation of the difficulty of school
transitions, (b) feelings of connectedness to the new
Although many questions remain, existing research school, and (c) persons who are perceived as most help-
seems to indicate that race and gender influence ful during the transition experience?
perceptions and outcomes of school transitions; how-
ever, the reason is unclear. One possibility might Method
concern a student’s level of engagement in school or Participants
connectedness to school. Eccles and colleagues (1993),
for example, suggested that school engagement might All of the students came from one middle school and
serve as a protective factor against transitional problems. one high school in a medium-sized, southeastern school
Also, Osterman’s (2000) review of research revealed that district. The middle school participants included 173 sixth-
students’ connectedness to school or sense of belonging grade students (approximately 72% of the sixth-grade
relates to a host of outcome variables including aca- class). The sample included 83 boys (48%), 86 girls (49.7%),
demic attitudes, achievement, attendance, participation and 4 students (2.3%) who did not provide information
in school activities, and dropping out of school. Thus, about gender. By race, the student sample included 57.2%
school connectedness is a variable that may affect the Caucasian (n ϭ 99), 19.7% African American (n ϭ 34),
likelihood of a successful school transition as well as 8.7% Asian American (n ϭ 15), 8.1% Latino (n ϭ 14), 4%
serve as an indicator of the actual success of that transi- multiracial (n ϭ 7), and 2.3% (n ϭ 4) unspecified. The
tion. Moreover, it seems likely that school transitions sample was reflective of the entire sixth-grade population
might undermine or disrupt students’ sense of connect- in terms of race (57% Caucasian, 23% African American,
edness to school, and this impact might not be equiva- 9% Asian American, 9% Latino, and 3% multiracial) and
lent across gender and race. gender (55% boys, 45% girls).
In addition to feelings of connectedness, students’ The high school sample comprised 320 ninth-grade
overall perceptions of the difficulty of the transition students (approximately 71% of the ninth-grade class) in
also seem to reflect their adjustment to a new school. a single high school. The sample included 47.8% boys
Thus, students’ assessment of the transition experience (n ϭ 153), 50.3% girls (n ϭ 161), and 1.9% unspecified
should provide insight into the extent to which they (n ϭ 6). The racial composition of the sample was 76.3%
have adapted to the new school. Moreover, the outcome Caucasian (n ϭ 244), 10.3% African American (n ϭ 33),
research reviewed in the previous paragraphs suggests 5.6% Asian American (n ϭ 18), 3.4% Latino (n ϭ 11),
that the difficulty of middle and high school transitions 2.2% multiracial (n ϭ 7), and 1.9% unspecified (n ϭ 6).
likely differs across gender and race. The research sample again represented the high school
population (n ϭ 1,586) in terms of race and gender.
Finally, if school transitions are difficult for many
students, who do the students find most supportive The middle school and high school were part of
during these times? Given that transitions often cause a medium-sized southern school district that included
a disruption of social networks at the very time they eight elementary schools, four middle schools, and two
are important (Barone, Aguirre-Deandreis, & Trickett, high schools. The middle school drew students primarily
1991) and that previous research indicates that school from three of the elementary schools; the high school
transition outcomes vary as a function of gender and attracted its students primarily from two of the middle
race, it seems likely that the persons who students schools. The middle school was in its first year of opera-
perceive as most helpful also would be influenced by tion. Overall, the school district could be characterized
566
as high performing—over 90% of the students attended the study. Homeroom or home-based teachers admin-
postsecondary education on a regular basis. istered the questionnaires; no incentives were given
for participation. Each questionnaire was precoded so
Instrument that school personnel could not identify student re-
sponses. Students returned questionnaires anonymously
We developed the questionnaires for the sixth grade in each classroom, and they were collected by a school
and ninth grade to tap context-relevant considerations counselor on site and delivered to the researchers. The
and student perceptions unique to each of the transi- research team worked with school personnel to gather
tions. The School Transition Questionnaire (STQ) is a demographic data for the participating students by pre-
retrospective measure of student perceptions over the coded numbers. Researchers matched gender and race
course of the transition (a copy of the STQ may be ob- to questionnaire responses by precoded identification
tained from the first author). The questionnaire assessed numbers.
a variety of information about the transition, including
students’ (a) overall feelings about the difficulty of the Research Design and Data Analysis
transition, (b) sense of connectedness to the new school,
and (c) persons who were most helpful to them during Data were analyzed separately for the middle and
the transition. high school samples. Gender and race represented
independent or predictor variables, whereas overall
We used a 4-point, Likert-type response format (e.g., perceptions of the transition, connectedness to school,
How was the move from middle school to high school and the person most helpful in the transition repre-
for you?) to capture students’ overall feelings about sented dependent or criterion variables in this causal–
the transition. The response choices were (1) difficult, comparative study. A 2 ϫ 4 (Gender ϫ Race) analysis
(2) somewhat difficult, (3) somewhat easy, and (4) easy. of variance (ANOVA) was planned for data analysis.
School connectedness is a variable that assesses a stu- Because of some low cell sizes, that analysis was contra-
dent’s integration and feelings of belonging to school. indicated. As a result, we completed separate univariate
Extensive research has demonstrated that school con- ANOVAs to analyze gender and race differences on the
nectedness is an important variable in school success variables. When we found significant F values, we used
(see Osterman, 2000 for a review of this literature). In post-hoc comparisons with Tukey’s Honestly Significant
this study, connectedness questions included feeling Differences Test to explain differences among racial
(a) close to other students, (b) a part of school, (c) that categories. Because we performed multiple tests to
teachers care about students, and (d) happy at school. determine who was most helpful in the transition, we used
We used a 5-point, Likert-type response format: re- the Bonferroni adjustment to control for Type 1 errors
sponse choices were (1) strongly disagree, (2) disagree, (a ϭ .05/3 ϭ .016).
(3) neither agree nor disagree, (4) agree, and (5) strongly
agree. The questions were adapted from the National Results
Longitudinal Study of Adolescent Health (1998), a study
undertaken in response to a mandate by the U.S. Con- Tables 1 and 2 present means and standard deviations for
gress in the NIH Revitalization Act of 1993. Coefficient the perceived difficulty of the transition, connectedness,
alpha for the 4-item connectedness measure in this study and helpful person variables disaggregated by gender
was .72 for the high school data and .71 for the middle
school data. and race.
Overall, the results revealed that students did not
Students also were assessed about perceived helpful-
ness of significant persons during the transition. Stu- perceive that the transition to middle school (M ϭ 3.00,
dents are influenced by a variety of persons in different SD ϭ .96) or high school (M ϭ 3.12, SD ϭ .86) was par-
systems in their life. Akos (2002) and Arowosafe and ticularly difficult (1 ϭ difficult to 4 ϭ easy). Students also
Irvin (1992) discovered that significant persons in the felt strongly connected in both middle school (M ϭ 15.02,
school environment provide support and information to SD ϭ 3.23) and high school (M ϭ 14.98, SD ϭ 2.60) to
students during school transitions. To determine who the new school after the transition (possible range, 4–20).
was most helpful in school adjustment, students rated a Parents (M ϭ 3.14, SD ϭ 1.03), followed by other stu-
variety of persons including parents, peers, other family dents (M ϭ 2.84, SD ϭ .98) and other family (M ϭ 2.44,
members, older students, counselors, and other adults SD ϭ 1.15), were reported as most helpful for middle school
at the school. The students used a 4-point, Likert-type students in transition. For high school students, other
scale: response choices were (1) not helpful, (2) some- students (M ϭ 2.77, SD ϭ .92), then parents (M ϭ 2.56,
what helpful, (3) helpful, and (4) very helpful. Students SD ϭ .99) and older students (M ϭ 2.34, SD ϭ 1.00) were
also had the option of adding persons to the list and rated as most helpful. The range was 1 ϭ not helpful to
rating them. 4 ϭ very helpful.
Procedure Gender as a Variable in School Transitions
The STQ was administered during the fall semester to Gender was not a significant variable in students’
all sixth- and ninth-grade students at the participating overall perception of the difficulty of the transition
schools. Approximately 72% of the middle school and (e.g., middle school girls, M ϭ 3.04, SD ϭ .93; middle
71% of the high school students chose to participate in school boys, M ϭ 2.95, SD ϭ 1.00; high school girls,
M ϭ 3.13, SD ϭ .90; high school boys, M ϭ 3.10, SD ϭ .83).
567
Table 1 Table 2
Disaggregated Means and Standard Deviations Disaggregated Means and Standard Deviations
for Perceptions of Difficulty, Connectedness, and for Perceptions of Difficulty, Connectedness, and
Helpful Others in Transition to Middle School Helpful Others in Transition to High School
Question M SD Question M SD
Overall difficulty 3.00 .96 Overall difficulty 3.12 .86
Boys 2.96 .11 Boys 3.08 .07
Girls 3.09 .10 Girls 3.13 .07
Caucasians 3.13 .16 Caucasians 3.14 .05
African Americans 3.15 .16 African Americans 3.26 .15
Latinos 2.00 .27 Latinos 2.78 .36
Asians 2.71 .29 Asians 2.69 .34
Multiracial students 2.86 .51 Multiracial students 2.83 .48
15.02 3.23 14.98 2.60
Connectedness 14.46 .41 Connectedness 15.65 .19
Boys 15.52 .32 Boys 14.43 .22
Girls 15.23 .29 Girls 15.04 .17
Caucasians 14.15 .69 Caucasians 15.16 .48
African Americans 14.63 1.67 African Americans 15.22 .52
Latinos 15.50 .73 Latinos 14.50 .58
Asians 14.86 1.61 Asians 14.50 .72
Multiracial students Multiracial students
3.14 1.03 2.56 .99
Helpful persons 2.44 1.03 Helpful persons 2.56 1.13
Parents 2.84 Parents 2.56
Other family 2.14 .98 Other family 2.56 .92
Students 2.23 1.08 Students 2.23 1.00
Older students 2.22 1.02 Older students 1.94
Counselors Teachers 1.91 .89
Other adults .99 Counselors .88
Other adults .87
Yet, gender was a significant variable in students’ feelings emerged for race in the perception of how difficult
of connectedness to school in both transitions. In middle the transition to middle school was, F(4, 160) ϭ 4.54,
school, girls (M ϭ 15.6, SD ϭ 2.7) felt more connected to p ϭ .002. Post-hoc tests (Tukey HSD) indicated that
school than did boys (M ϭ 14.4, SD ϭ 3.6), F(l, 157) ϭ 4.59, Latino students (M ϭ 2.07, SD ϭ .25) perceived the
p ϭ .034. In contrast to middle school, boys (M ϭ 15.7, transition as more difficult as compared with Caucasian
SD ϭ 2.2) felt significantly more connected in high school (M ϭ 3.12, SD ϭ .09, p ϭ .001) and African American
than did girls (M ϭ 14.4, SD ϭ 2.8), F(l, 300) ϭ 18.52, (M ϭ 3.15, SD ϭ .16, p ϭ .003) students.
p ϭ .001.
Also in middle school, several differences were found
Gender also was a significant variable in determin-
ing who was most helpful during the transition to high for persons who were most helpful in the transition. Sta-
school. Significant differences emerged for family other
than parents, F(l, 302) ϭ 13.07, p Ͻ .001, and stu- tistically significant differences emerged for race in terms
dents, F(l, 303) ϭ 18.61, p Ͻ .001. Boys reported that of middle school counselors, F(4, 153) ϭ 7.48, p Ͻ .001,
family other than parents (M ϭ 2.55, SD ϭ 1.13) and and family other than parents, F(4, 156) ϭ 3.81, p ϭ .006.
students (M ϭ 3.00, SD ϭ .84) were more helpful than In post-hoc tests (Tukey HSD), Latino students reported
that reported by girls (M ϭ 2.09, SD ϭ 1.08; M ϭ 2.56, that middle school counselors (M ϭ 3.24, SD ϭ.83) and
SD ϭ .94) in high school. No other significant differences family other than parents (M ϭ 3.46, SD ϭ .66) were
were revealed for gender. more helpful than that reported by Caucasian (M ϭ 1.95,
SD ϭ .92 for counselors; M ϭ 2.32, SD ϭ 1.15 for other
Race as a Variable in School Transitions family) and Asian American (M ϭ 2.00, SD ϭ .88 for
counselors; M ϭ 2.20, SD ϭ .94 for other family) students.
No significant differences were found for race in feelings In addition, Latino students reported that other family
of connectedness to school. Yet, a significant difference members (M ϭ 3.46, SD ϭ .66) were more helpful than
568 that reported by African American students (M ϭ 239,
SD ϭ 1.20). Finally, African American students reported
that middle school counselors (M ϭ 2.61, SD ϭ 1.09) apply themselves as fully academically as do girls. At
were more helpful to them as compared with reports by the same time, connectedness may be an adjustment
Caucasian students (M ϭ 1.95, SD ϭ .92). variable that does not capture the decline in self-esteem
of girls in transition. Those relationships should be ex-
In high school, significant differences also were amined in future research.
found for race for help from high school counselors,
F (5, 297) ϭ 3.54, p ϭ .004. Latino students (M ϭ 2.80, In contrast to middle school transition, boys felt signif-
SD ϭ .27) reported that high school counselors were icantly more connected than did girls after the transition
more helpful compared with reports by Caucasian stu- to high school. Although that finding may reflect a cohort
dents (M ϭ 1.86, SD ϭ .06). No other significant differ- effect for the current sample, the results also may indicate
ences were found for race in terms of perceptions or more positive adaptation or orientation for boys at the
who was helpful during the transition for the middle or high school level. Fewer researchers have investigated
high school sample. gender differences in outcomes for high school transi-
tion as compared with middle school transition. Yet,
Discussion these findings may be reflective of Maute’s (1991) find-
ing that girls exhibit more concerns in general and more
Whereas Eccles and colleagues (1993), Seidman and col- intense social and academic concerns than do boys in
leagues (1994), and Simmons and Blyth (1987) reported the transition to high school. Although the connection to
negative outcomes for school transitions, students in school is strong over the transition to middle school, the
this study perceived the transition as somewhat easy. peer upheaval and psychological distress experienced by
Study participants also demonstrated a strong connec- girls in the transition to and over the course of middle
tion to school and found a variety of school personnel school (Blyth et al., 1983; Crockett et al., 1989; Eccles
and other persons helpful during school transitions. The et al., 1993; Fenzel, 1989) may present later in their feel-
contextual nature of the school district (e.g., high per- ing less connected after the transition to high school.
forming) may be reflected in the perceived difficulty of Researchers need to examine the relationship between
the transition in this study because our results support ecological factors (e.g., extracurricular opportunities and
those from previous research (Anderman, Maeher, & participation); feelings of connectedness, self-esteem,
Midgley, 1999; Crockett, Peterson, Graber, Schulenberg, & achievement, and motivation; as well as effective ways
Ebata, 1989) that suburban students experience fewer to connect students in the transition to a new school.
adverse effects in school transitions than do urban
students. Implications of race and gender as variables One common way to help students connect to school
in transition adjustment are discussed in the following may be through enhanced social support. Barone and
paragraphs. colleagues (1991) suggested that that type of support
may be particularly important during transition because
Gender social networks are usually disrupted and in flux at this
time. The findings for connectedness in this study may be
The results of this study demonstrate that girls felt more indicative of the gender differences in who appears help-
connected to school than did boys after transition to mid- ful during school transitions. Whereas girls reported that
dle school. Although the stronger connection to school family other than parents were more helpful at their tran-
may suggest more positive adaptation or orientation to sition into middle school, the finding was reversed when
middle school for girls than for boys, these findings seem they entered high school. Even more significant, boys
incongruent with previous research (Blyth et al., 1983; reported that students were significantly more helpful
Eccles et al., 1993) stating that self-esteem declines for in the transition to high school than girls reported. Elias
girls. Therefore, these findings raise several questions: and colleagues’ (1992) findings of the salience of peer
Do girls experience a stronger connection to middle networks in middle school and Chung and colleagues’
school because of the contextual nature of elementary (1998) findings of more peer-related problems for girls
and middle schools? For example, does the presence of in middle school may culminate in lower perceived sup-
multiple female models in elementary and middle school port from peers during the transition to high school. That
allow girls to feel more connected to school? Academic lack of perceived social support also may relate to lower
outcomes were not measured in this study, but poten- feelings of connectedness to school and highlight an
tially, feelings of connectedness may relate directly to important developmental need for girls when they move
academic outcomes in transition. The lower feelings of to high school. Programs like peer mentoring, use of
connectedness for boys in this study may provide a pos- students in transition programs, and building supportive
sible explanation for the findings of previous studies peer cultures may be instrumental in helping girls feel
that suggest that boys suffer more distinct academic de- more connected to high school.
clines in the transition to middle school (Chung, Elias, &
Schneider, 1998; Elias et al., 1992). Race
Osterman (2000) demonstrated that a relationship ex- Latino students perceived that the transition to middle
ists between connectedness or belonging and academic school was significantly more difficult than did Cauca-
outcomes. Perhaps boys’ academic struggles caused by sian and African American students. The results of this
the transition to middle school are related to feelings study may be reflective of Wampler and colleagues’
of connectedness to the new school. Boys may feel less (2002) finding that Latino students experience a sig-
connected to the new school and, therefore, might not nificant decline in GPA during the transition to middle
569
school. Although GPA was not examined in this study, negative outcomes of transitions, perhaps Latino stu-
the overall perception of the transition difficulty may re- dents are unique in that learning, and adjusting to the
flect or be an indicator of difficult academic adjustment U.S. education system contributes to a less difficult expe-
demonstrated by Latinos in previous research. One also rience in the second transition. A more likely possibility,
should consider the findings (perception of difficulty of however, with the graduation rate of Latino students
transition) and previous research (GPA declines) in light near 57% (Narrid-Lacey & Spencer, 2000), is that many
of cultural differences that may create specific challenges of the Latino students who would have experienced
for Latino students in transition. difficulty with the high school transition have already
dropped out of school. Future research is clearly needed
The results of this study and previous research to understand the specific transition effects for Latino
(Wampler et al., 2002) suggest that Latino students, many students in middle and high school.
of whom were not born in this country, may need par-
ticular attention during the transition to middle school. Limitations
Wampler and colleagues speculated that peer culture
pressures and values and traditions in Latino culture The results of this study should be considered in light of
might increase the struggles that those students experi- several limitations. First, data were collected at only one
ence in transition. For example, Latino students may point after the completion of the transition. We exam-
experience more significant language barriers, and Latino ined students’ adjustment to a new level of school, but
families may believe that it is not appropriate to be in- longitudinal designs may capture variables that influence
volved in matters related to school (McCall-Perez, 2000). adjustment prior to the transition. Also, one should inter-
In addition, cognitive and linguistic demands are greater pret the data as relevant to the context of the academic
in secondary schools as compared with elementary achievement of students in a district in which a large ma-
schools. In secondary schools, students must understand jority progressed to postsecondary education. Researchers
and integrate bodies of knowledge (Lucas, 2000). The should replicate similar procedures in a variety of districts
gap between language and literacy skills between Latino to determine whether race and gender effects are common
students and families and majority teachers may create across a variety of school contexts. Although no district is
additional barriers to achievement in the more demand- representative of all districts, our results may be particu-
ing secondary schools (Osterling, Violand-Sanchez, & larly important because of the intense academic expecta-
von Vacano, 1999). tions of students in the participating district. Learning how
to facilitate successful school transitions and school adjust-
The cultural effects found in this study also may be ment for Latino students in the midst of high academic
reflected in the fact that the Latino students reported that expectations may be extremely informative and useful for
middle school counselors and family other than parents promoting achievement for all students.
were more helpful than that reported by Caucasian
and Asian students. McCall-Perez (2000) reported that Conclusion
students with limited English proficiency are counselor
dependent for school success because counselors con- To summarize, the results of this study support previ-
tribute to more positive student outcomes, placement, ous research that suggests that gender and race are
credit hours earned, and smoother transitions to high influential variables in school transitions and highlight
school. It also may be useful to include extended fam- potential differences in transition programming needed
ily to help facilitate student transition and adjustment for different groups of students. Therefore, it may be
to middle and high school. For example, schools might important to consider gender in assisting students’ ad-
offer bilingual evening transition programs with child- justment to school. Although orientation programs may
care to enable families and school personnel to help provide the necessary procedural (e.g., how to register
facilitate Latino students’ transition. Schools also might for classes) and organizational adjustment (e.g., how to
offer programs delivered by diverse paraprofessionals navigate the new school) assistance for all students, re-
who can bridge language and cultural barriers effectively searchers who investigate transition programs may need
(Lee, 2001). Lucas (2000) recommended that school per- to attend to gender differences in specific needs regard-
sonnel should help Latino students and families under- ing personal/social and academic adjustment. Perhaps
stand and negotiate the U.S. system of education (e.g., those researchers should further examine the needs of
grading, programs, placement, parent involvement). boys when they transfer to middle school so as to foster
stronger feelings of connectedness (e.g., extracurricular
In contrast to the middle school findings, race was activities, male models) that may help prevent boys’
not a significant factor in the perception of transition dif- achievement declines. Similarly, researchers should fur-
ficulty. Positive interventions by counselors and others ther investigate girls’ needs when they transfer to high
during the transition and/or the small sample of Latino school to build stronger feelings of connectedness.
students in the high school might have made differences Perhaps education professionals can facilitate more
difficult to detect. It also is possible that Latino students effective personal/social adjustment to high school with
might recover from initial difficulties in the transition to programs like peer mentoring and systematic efforts
middle school and feel more successful by the time they to improve female peer cultures so that girls also find
make the transition to high school. Although Blyth and other students helpful during transition.
colleagues (1983) suggested that multiple transitions
(e.g., elementary to middle to high school) exacerbate
570
It also may be useful to offer specific programming Diemert, A. (1992). A needs assessment of fifth grade stu-
for Latino students and families. The transition may be dents in a middle school. Acton, MA: Author. (ERIC
an influential factor in Latino students’ success in school; Document Reproduction Service No. ED62332)
research has demonstrated that student engagement is
a predictor of achievement for Latino students (Lucas, Eccles, J. S., Wigfield, A., Midgley, C., Reumam, D.,
2000; Wampler et al., 2002). The transition to middle Mac Iver, D., & Feldlaufer, J. (1993). Negative ef-
school may be a key point in the developmental trajec- fects of traditional middle schools on students’
tory of Latino students that can be used to prevent high motivation. The Elementary School Journal, 93,
rates of educational failure and dropping out. Finally, it 553–574.
is important that students capitalize on social support in
the transition to a new school. Specifically in this study, Elias, M., Ubriaco, M., Reese, A., Gara, M., Rothbaum, P., &
school counselors, other students, and family other than Haviland, M. (1992). A measure of adaptation
parents appear to be most useful in helping minority to problematic academic and interpersonal tasks
students adjust to school. of middle school. Journal of School Psychology, 30,
41–57.
NOTE
Fenzel, L. (1989). Role strains and the transition to
We thank the Research Triangle Schools Partnership middle school: Longitudinal trends and sex differ-
(RTSP) and the Chapel Hill Carrboro City Schools ences. Journal of Early Adolescence, 9, 211–226.
for financial assistance with this research. We
appreciate Pat Harris, Tori Lunetta, and Annie Gutman, L., & Midgley, C. (2000). The role of protec-
Reed for invaluable feedback and help with the tive factors in supporting the academic achieve-
research. We also thank Laura Blake, Kelley Dull, ment of poor African American students during
Jessica Thompson, and Sarah Doherty for preparing the middle school transition. Journal of Youth and
questionnaires and inputting the data. Adolescence, 29, 223–248.
REFERENCES Hirsch, B., & Rapkin, B. (1987). The transition to junior
high school: A longitudinal study of self-esteem,
Akos, P. (2002). The developmental needs of students psychological symptomology, school life, and social
in transition from elementary to middle school. support. Child Development, 58, 1235–1243.
Professional School Counseling, 5, 339–345.
Lee, C. (2001). Culturally responsive school counsel-
Anderman, E., Maeher, M., & Midgley, C. (1999). ors and programs: Addressing the needs of all stu-
Declining motivation after the transition to middle dents. Professional School Counseling, 4, 257–261.
school: Schools can make a difference. Journal
of Research and Development in Education, 32, Lucas, T. (2000). Facilitating the transitions of second-
131–147. ary English language learners: Priorities for Princi-
pals. NASSP Bulletin, 84, 2–16.
Arowosafe, D., & Irvin, J. (1992). Transition to a
middle level school: What kids say. Middle School Maute, J. (1991). Transitions concerns of eighth-grade
Journal, 24, 15–19. students in six Illinois schools as they prepare for
high school. Unpublished doctoral dissertation,
Barone, C., Aguirre-Deandreis, A., & Trickett, E. (1991). National-Louis University, Evanston, IL.
Mean-ends problem-solving skills, life stress, and
social support as mediators of adjustment in the McCall-Perez, Z. (2000). The counselor as advocate
normative transition to high school. American Jour- for English language learners: An action research
nal of Community Psychology, 19, 207–225. approach. Professional School Counseling, 4,
13–20.
Blyth, D., Simmons, R., & Carlton-Ford, S. (1983). The
adjustment of early adolescents to school transi- Mosely, J., & Lex, A. (1990). Identification of potentially
tions. Journal of Early Adolescence, 3, 105–120. stressful life events experienced by a population
of urban minority youth. Journal of Multicultural
Cauce, A., Hannan, K., & Sargeant, M. (1992). Life Counseling and Development, 18, 118–125.
stress, social support, and locus of control dur-
ing early adolescence: Interactive effects. American Narrid-Lacey, B., & Spencer, D. (2000). Experiences of
Journal of Community Psychology, 12, 353–367. Latino immigrant students at an urban high school.
NASSP Bulletin, 84, 43–54.
Chung, H., Elias, M., & Schneider, K. (1998). Patterns
of individual adjustment changes during middle National Longitudinal Study of Adolescent Health.
school transition. Journal of School Psychology, 36, (1998). Codebooks. Retrieved June 19, 2002, from
83–101. http://www.cpc.unc.edu/addhealth/codebooks.html
Crockett, L., Peterson, A., Graber, J., Schulenberg, J., & Osterling, J., Violand-Sanchez., E., & von Vacano, M.
Ebata, A. (1989). School transitions and adjustment (1999). Latino families learning together. Educa-
during early adolescence. Journal of Early Adoles- tional Leadership, 57, 64–68.
cence, 9, 181–210.
Osterman, K. (2000). Students’ need for belonging
in the school community. Review of Educational
Research, 70, 323–367.
Seidman, E., Allen, L., Aber, J., Mitchell, C., &
Feinman, J. (1994). The impact of school transi-
tions in early adolescence on the self-system and
perceived social context of poor urban youth. Child
Development, 65, 507–522.
571
Simmons, R., & Blyth, D. (1987). Moving into adoles- tion to junior high. Journal of School Psychology,
cence: The impact of pubertal change and school 40, 213–237.
context. Hawthorne, NY: Aldine de Gruyter. Wigfield, A., Eccles, J., Mac Iver, D., Reuman, D., &
Midgley, C. (1991). Transitions during early ado-
Simmons, R., Black, A., & Zhou, Y. (1991). African- lescence: Changes in children’s domain specific
American versus white children and the transition self-perceptions and general self-esteem across the
into junior high school. American Journal of Edu- transition to junior high school. Developmental Psy-
cation, 99, 481–520. chology, 27, 552–566.
Wampler, R., Munsch, J., & Adams, M. (2002). Ethic
differences in grade trajectories during the transi-
Copyright of Journal of Educational Research is the property of Heldref Publications and its content may not be copied or emailed to
multiple sites or posted to a listserv without the copyright holder’s express written permission. However, users may print, download,
or email articles for individual use.
Source: Akos, P. and Galassi, J. P. (2004). Gender and Race as Variables in Psychosocial Adjustment to Middle and High School. The
Journal of Educational Research, (98)2, 102–109.
572
Gender and Race as Variables in Psychosocial Adjustment
to Middle and High School
Self-Test for Task 11
Y ϭ Yes
N ϭ No
NA ϭ Not applicable
?/X ϭ Can’t tell/Don’t know
General Evaluation Code
Introduction ______________
Problem ______________
______________
Is there a statement of the problem? Does the problem indicate ______________
a particular focus of study?
Is the problem researchable; that is, can it be investigated by ______________
collecting and analyzing data? ______________
Is background information on the problem presented?
Is the educational significance of the problem discussed? ______________
Does the problem statement indicate the variables of interest
and the specific relation among the variables that were
investigated?
When necessary, are variables directly or operationally defined?
Did the researcher have the knowledge and skill to carry out
the research?
Review of Related Literature ______________
Is the review comprehensive? ______________
Is the review well organized? Does it logically flow in such ______________
a way that the references least related to the problem are
discussed first and those most related are discussed last? Does ______________
it educate the reader about the problem or topic? ______________
______________
Is the review more than a series of abstracts or annotations? ______________
That is, have the references been analyzed and critiqued and ______________
the results of various studies compared and contrasted? ______________
Have the references been critically analyzed and the results of
various studies compared and contrasted (i.e., is the review
more than a series of abstracts or annotations)?
Are all cited references relevant to the problem under
investigation? Is the relevance of each reference explained?
Does the review conclude with a summary and interpretation of
the literature and its implications for the problem under study?
Do the implications form an empirical or theoretical rationale
for the hypotheses that follow?
Are most of the sources primary (i.e., are there only a few or
no secondary sources)?
Are references cited completely and accurately?
Hypotheses ______________
Are specific research questions listed or specific hypotheses ______________
stated?
Is each hypothesis testable? 573
Does each hypothesis state an expected relation or difference? ______________
If necessary, are variables directly or operationally defined? ______________
Method ______________
______________
Participants ______________
Are the size and major characteristics of the population
described? ______________
Are the accessible and target populations described? ______________
If a sample was selected, is the method of selecting the sample
clearly described? ______________
Does the method of sample selection suggest any limitations
or biases in the sample? For example, was stratified sampling
used to obtain sample subgroups?
Are the size and major characteristics of the sample described?
If the study was quantitative, does the sample size meet the
suggested guidelines for the minimum sample size appropriate
for the method of research represented?
Instruments ______________
Do instruments and their administration meet guidelines for ______________
protecting participants? Were needed permissions obtained?
______________
Are the instruments appropriate for measuring the intended ______________
variables? ______________
______________
Was the correct type of instrument used for data collection
(e.g., was a norm-referenced instrument used when a ______________
criterion-referenced one was more suitable)?
______________
Is the rationale given for the selection of the instruments
(or measurements) used? ______________
______________
Are the purpose, content, validity, and reliability of each
instrument described?
If appropriate, are subtest reliabilities given?
Is evidence presented to indicate that the instruments are
appropriate for the intended sample? For example, is the read-
ing level of an instrument suitable for sample participants?
If an instrument was developed specifically for the study, are
the procedures involved in its development and validation
described?
If an instrument was developed specifically for the study, are
administration, scoring or tabulating, and interpretation
procedures fully described?
Does the researcher have the needed skills or experience to
construct or administer an instrument?
Design and Procedure
Are the design and procedures appropriate for examining ______________
the research question or testing the hypotheses of the study?
Are the procedures described in sufficient detail to permit ______________
replication by another researcher? ______________
Do procedures logically relate to one another?
Were instruments and procedures applied correctly? ______________
If a pilot study was conducted, are its execution and results
described? ______________
574 Is the effect on the subsequent study explained? ______________
Are control procedures described? ______________
______________
Does the researcher discuss or account for any potentially
confounding variables that he or she was unable to control?
Results ______________
______________
Are appropriate descriptive statistics presented? ______________
Are the tests of significance appropriate, given the hypotheses ______________
and design of the study? ______________
______________
If parametric tests were used, is there evidence that the ______________
researcher avoided violating the required assumptions for ______________
parametric tests? ______________
Was the probability level at which the tests of significance were
evaluated specified in advance of the data analyses? Was every
hypothesis tested?
Are the tests of significance interpreted using the appropriate
degrees of freedom?
Was the inductive logic used to produce results in a qualitative
study made explicit?
Are the results clearly described?
Are the tables and figures (if any) well organized and easy
to understand?
Are the data in each table and figure described in the text?
Discussion (Conclusions and Recommendations) ______________
Is each result discussed in terms of the original hypothesis ______________
or topic to which it relates? ______________
______________
Is each result discussed in terms of its agreement or ______________
disagreement with previous results obtained by other ______________
researchers in other studies?
______________
Are generalizations consistent with the results?
Are theoretical and practical implications of the findings
discussed?
Are the possible effects of uncontrolled variables on the results
discussed?
Are recommendations for future action made?
Are the suggestions for future action based on practical
significance or on statistical significance only (i.e., has the
author avoided confusing practical and statistical significance)?
Abstract or Summary ______________
Is the problem stated? ______________
Are the number and type of participants and instruments ______________
described? ______________
Is the design identified? ______________
Are procedures described?
Are the major results and conclusions stated?
Type-Specific Evaluation Criteria ______________
575
Are the characteristics or experiences that differentiate the
groups (i.e., the grouping variable) clearly defined or described?
Were any control procedures applied to equate the groups on ______________
extraneous variables? ______________
Are causal relations discussed with due caution? ______________
Are plausible altrenative hypotheses discussed?
576
APPENDIX A
Reference Tables
Table A.1 Ten Thousand Random Numbers
Table A.2 Values of the Correlation Coefficient for Different Levels of Significance
Table A.3 Standard Normal Curve Areas
Table A.4 Distribution of t
Table A.5 Distribution of F
Table A.6 Distribution of x2
577
578 APPENDIX A • REFERENCE TABLES
TABLE A.1 • Ten thousand random numbers
APPENDIX A • REFERENCE TABLES 579
TABLE A.1 • Continued
580 APPENDIX A • REFERENCE TABLES
TABLE A.1 • Continued
APPENDIX A • REFERENCE TABLES 581
TABLE A.1 • Continued
The words you are searching are inside this book. To get more targeted content, please make full-text search by clicking here.
E Book Educational Research L R Gay Pearson 2012
Discover the best professional documents and content resources in AnyFlip Document Base.
Search
E Book Educational Research L R Gay Pearson 2012
- 1 - 50
- 51 - 100
- 101 - 150
- 151 - 200
- 201 - 250
- 251 - 300
- 301 - 350
- 351 - 400
- 401 - 450
- 451 - 500
- 501 - 550
- 551 - 600
- 601 - 650
- 651 - 667
Pages: