Assessment Based in Evidence
and Law:
ABEL for Practicing Clinicians
ASHA Convention 2006
Miami
Catherine J. Crowley, MA, CCC-SLP, JD, [email protected]
Natalie Agosto, MS, CF [email protected]
George Castle, MS, CF [email protected]
Lindsay Friedman, MS, CF [email protected]
Pamela S. Ouellette, MPA [email protected]
Dorothy Tancredi, MS, CF [email protected]
Copyright Crowley, Agosto, Castle, Friedman, 1
Ouellette, Tancredi 2006
Where it all began. . .
ADVANCE column suggested by students
in the Assessment and Evaluation
course, Teachers College Columbia
University, Fall 2005.
Students in field having supervisors who
did not have the up to date knowledge
base.
Copyright Crowley, Agosto, Castle, Friedman, 2
Ouellette, Tancredi 2006
Why the ADVANCE?
Survey of approximately 800 practicing
SLPs over one year (2004-2005).
Over 90% said that they read the
ADVANCE.
The ADVANCE for SLP has over 60,000
subscriptions and over half the
subscribers work in the schools.
Copyright Crowley, Agosto, Castle, Friedman, 3
Ouellette, Tancredi 2006
Review of the Columns
The federal law: IDEA 2004
Current research on construct validity:
Discriminant accuracy standard (Plante and
Vance (1994))
Statistical accuracy of test scores:
Confidence intervals
Copyright Crowley, Agosto, Castle, Friedman, 4
Ouellette, Tancredi 2006
Column Review (con’t)
The parent interview
The teacher interview
Identifying and addressing
cultural and linguistic biases in
assessment materials.
Copyright Crowley, Agosto, Castle, Friedman, 5
Ouellette, Tancredi 2006
Case Study—Abel, 4-year old boy
Referred for an evaluation by his preschool
teacher
reported that his speech is unclear
reported that he communicates primarily using
gestures or non-specific one-word utterances
Pulled him out of arts & crafts activity for
evaluation
Abel grabbed my digital recorder and began
spinning around the room singing unintelligibly
Abel refused to answer questions from PLS-4
Abel cried inconsolably when I tried to read a book
Abel threw my planned activities on the floor and
repeatedly tried to leave the room
Copyright Crowley, Agosto, Castle, Friedman, 6
Ouellette, Tancredi 2006
What Should I Do?
Report that Abel
is immature, impulsive, unfocused, and uncooperative?
expresses needs and wants through bad behavior
rather than through language?
has unintelligible speech?
has poor eye contact and topic maintenance?
refused to participate in the evaluation?
NO! I need to re-assess my evaluation methodology!
My first stop, should have been to speak with Abel’s
primary caregiver.
Copyright Crowley, Agosto, Castle, Friedman, 7
Ouellette, Tancredi 2006
ABEL Methodology
The Primary Caregiver Interview
A child’s primary caregivers have the critical
information their child’s:
communication skills across communicative contexts
and partners
cultural background
exposure to various dialects and languages
experience with literacy and play
Every primary caregiver should be asked 7 critical
questions . . .
Copyright Crowley, Agosto, Castle, Friedman, 8
Ouellette, Tancredi 2006
Critical Question #1
What is the highest educational level of the
mother or primary caregiver?
Research has showed that the child’s vocabulary and
literacy achievement is positively correlated with the
mother’s educational level.
Ex. Abel’s mother, who is bilingual, returned to
community college for an associate’s degree in
radiological imaging and his father, who is a
monolingual Spanish speaker works for a dry cleaner
and a restaurant.
Copyright Crowley, Agosto, Castle, Friedman, 9
Ouellette, Tancredi 2006
Critical Question #2
Is there a family history of speech, language,
and/or academic problems? (Restrepo, 1998)
A genetic basis exists for many communication
disorders.
Ex. Abel’s cousin, who is 8 y.o., is in special education
classes at his public school. He is reportedly “slow.”
Copyright Crowley, Agosto, Castle, Friedman, 10
Ouellette, Tancredi 2006
Critical Question #3
How does how the child’s speech and language
development compare to his/her siblings at
the same age or to peers in the child’s speech
community? (Restrepo, 1998)
The key consideration in distinguishing between a
language difference and a language disorder are the
language norms of the child’s speech community.
Ex. Abel’s speech, language, and motor skills
developed more slowly than his sister Lillith and his
brother Cain. Abel didn’t say his first word until age
2;2. Abel didn’t walk until 20 months. Abel’s mother
says he is “different” from her other children. He is
the baby of the family.
Copyright Crowley, Agosto, Castle, Friedman, 11
Ouellette, Tancredi 2006
Critical Question #4
Was the child’s performance during the
evaluation typical?
A child may not want to “perform” for a stranger
seated at a tiny table in a small room with boring
spiral-bound manuals full of pictures of watches and
rubber ducks
Ex. Abel’s mother says he talks non-stop to anyone
who will listen, although not many people understand
what he says. She worries that he is not afraid of
strangers.
Copyright Crowley, Agosto, Castle, Friedman, 12
Ouellette, Tancredi 2006
Critical Question #5
Have there been any significant changes in the
family structure recently?
the child’s performance during the evaluation may
be affected by factors outside of the evaluator’s
(and the child’s) control.
birth of a sibling
death in the family
serious illness
divorce
Ex. Abel’s father resents that his wife is completing
a college degree while he often works 100 hours a
week at two low-paying jobs. They fight when
dinner is not on the table or the laundry is not
done.
Copyright Crowley, Agosto, Castle, Friedman, 13
Ouellette, Tancredi 2006
Critical Question #6
What exposure has your child had to different
languages or dialects?
a child’s language may be evidence of a language
disorder or . . .
second language acquisition (including a silent
period)
subtractive bilingualism
correct dialect forms
code-switching
Ex. Abel’s parents speak primarily Spanish at home
and in the community. Abel’s brother and sister code-
switch between Spanish, Spanish-influenced English,
and African American Vernacular English.
Copyright Crowley, Agosto, Castle, Friedman, 14
Ouellette, Tancredi 2006
Critical Question #7
What does your child do that makes you know
(s)he is smart?
provides insight into the child’s abilities,
provides insight into the parent’s reliability as a
reporter
especially useful when the child is a multiply disabled
child or an adolescent.
Ex. When his father has difficulty communicating in
English, Abel tries to translate for him, but Abel is not
always accurate, nor is he always understood by his
father or by others.
Copyright Crowley, Agosto, Castle, Friedman, 15
Ouellette, Tancredi 2006
What Did I Learn About Abel?
Abel’s parents are not college-educated professionals
Abel may not have had home exposure to the linguistic and
academic skills necessary for success in the mainstream
middle-class American classroom.
Abel is identified as being “different” from his siblings in
terms of his linguistic and motoric development.
A family history of a learning disability may exist.
But—Abel’s siblings are proficient speakers of two
languages and of two dialects of English.
Abel is a helpful, gregarious child who enjoys talking to and
interacting with strangers
Abel tries to translate between Spanish and English for his
monolingual Spanish-speaking father.
Abel’s family is experiencing a great deal of tension 16
surrounding his mother’s decision to pursue her education
and his father’s need to work two low-paying, tiring jobs.
Copyright Crowley, Agosto, Castle, Friedman,
Ouellette, Tancredi 2006
The Teacher Interview
Critical Question #1
What is student’s grade level
achievement in reading and
math? Provides quantifiable data.
Copyright Crowley, Agosto, Castle, Friedman, 17
Ouellette, Tancredi 2006
The Teacher Interview
Critical Question #2
What supports does he need?
What are student’s strengths
and weaknesses?
Copyright Crowley, Agosto, Castle, Friedman, 18
Ouellette, Tancredi 2006
The Teacher Interview
Critical Question #3
How has he progressed over
time? Do you have a portfolio or
examples of his work?
Copyright Crowley, Agosto, Castle, Friedman, 19
Ouellette, Tancredi 2006
The Teacher Interview
Critical Question #4
How do his language skills
compare to those of his
classmates?
Copyright Crowley, Agosto, Castle, Friedman, 20
Ouellette, Tancredi 2006
The Teacher Interview
Critical Question #5
Does the teacher concur with
the conclusions the SLP
reached as a result of the
evaluation?
Copyright Crowley, Agosto, Castle, Friedman, 21
Ouellette, Tancredi 2006
The Teacher Interview
Self Question:
Does it seem that the student is
receiving an adequate
education based upon the
teacher interview?
Copyright Crowley, Agosto, Castle, Friedman, 22
Ouellette, Tancredi 2006
The Teacher Interview
A child cannot be found to have a
disability if the determinant factor
for the disability is:
“lack of appropriate instruction in
reading, including the essential
components of reading instruction”, or
“lack of instruction in math.”
20 U.S.C. § 1414(b)(5)(A)(B)
Copyright Crowley, Agosto, Castle, Friedman, 23
Ouellette, Tancredi 2006
Cultural and Linguistic Biases in
Standardized Tests
IDEA 2004: “No cultural or racial biases”
Despite review efforts, potential for biases
remain
“Home” and “school” languages can influence
standardized test results.
“Rather than measuring what they claim to
measure, test items sometimes measure
instead knowledge of standard forms of the
language (Wolfram, 1999).”
Copyright Crowley, Agosto, Castle, Friedman, 24
Ouellette, Tancredi 2006
Cultural and Linguistic Bias
as a Result of Census Data
Census data used for standardization samples
show the average performance of all represented
groups
The Issue
Many standardized tests have poor
representation of minorities using the 2000
U.S. Census data.
The average of the entire group may not
reflect the average of individual speech
communities
Copyright Crowley, Agosto, Castle, Friedman, 25
Ouellette, Tancredi 2006
A Popular Preschool
Spanish Exam…
Tool to identify
monolingual/bilingual Spanish
speakers with a disorder or delay.
Cultural and linguistic biases
compromise validity of test.
Adapted Spanish tests have over-
representation of one regional/
cultural group.
Copyright Crowley, Agosto, Castle, Friedman, 26
Ouellette, Tancredi 2006
Standardization Sample
Of 1,188 children in the standardization
sample:
87% spoke a dialect of Mexican Spanish
29 (2.4%) Cuban children included
33 (2.8%) Puerto Rican children included
9 (.8%) Colombian children included
2 (.2%) Ecuadorian children included
* Spanish proficiency varied and should have been
clustered around a mean
Copyright Crowley, Agosto, Castle, Friedman, 27
Ouellette, Tancredi 2006
Puntiagudas vs Dedo
Gordo
Thumb 28
Copyright Crowley, Agosto, Castle, Friedman,
Ouellette, Tancredi 2006
What if…
What if this exam was administered to
ABEL when he was 4 years old?
Where would he fall into this normative
sample?
Exposed to two dialects of Spanish
and English
Mother: Colombian
Father: Ecuadorian
Copyright Crowley, Agosto, Castle, Friedman, 29
Ouellette, Tancredi 2006
Cultural and Linguistic
Biases
Normative sample falls short in:
Discriminant accuracy
Sensitivity and Specificity
Ability to predict whether ABEL’s
performance falls within the norms of
his speech community
Copyright Crowley, Agosto, Castle, Friedman, 30
Ouellette, Tancredi 2006
Cultural and Linguistic
Biases
Children’s test performance is affected by
prior knowledge and experience
Language socialization practices:
Labeling
Exposure to books/Book reading styles
Known questions
Copyright Crowley, Agosto, Castle, Friedman, 31
Ouellette, Tancredi 2006
Labeling
Item/picture labeling vs. functional
description
Children from labeling cultures may
perform better on standardized
vocabulary tests
Children from non-labeling cultures may
perform worse on standardized tests but
have a deeper semantic knowledge of
targeted vocabulary words
Copyright Crowley, Agosto, Castle, Friedman, 32
Ouellette, Tancredi 2006
Book Reading Styles
According to Anderson-Yockel & Haynes
(1994), European American mothers asked
significantly more yes/no and “wh” questions
than did African American mothers.
European American Dyads appeared to be
more consistent with mainstream school
cultural expectations which rely on
questioning strategies to demonstrate
knowledge and literacy.
Copyright Crowley, Agosto, Castle, Friedman, 33
Ouellette, Tancredi 2006
Known Questions
According to Langdon and Cheng (1992),
Latino children are not typically asked to retell
events or facts that adults already know.
Ex. “Tell me about the fire engine on the
table.”
Copyright Crowley, Agosto, Castle, Friedman, 34
Ouellette, Tancredi 2006
The Sociolinguistic Parameters of:
Topic 35
Participant
Setting
Function
Copyright Crowley, Agosto, Castle, Friedman,
Ouellette, Tancredi 2006
Holograms
Make student come to life through examples
Ex. (e.g. “ Initially, Abel had marked difficulty with syllable
segmentation. Using Dynamic Assessment, the examiner
explained syllable segmentation by clapping his hands to
each syllable in a word. Later, during examples 15 through
20 of the phonemic awareness section, Abel looked up
towards the ceiling and segmented the 3 syllable words by
slicing his hand through the air as he said each syllable out
loud (E-le-phant). This demonstrates Abel’s learning
potential as he seemed to learn segmentation after minimal
modeling. It also showed that he developed his own
strategy, slicing his had instead of clapping. ”)
*For further information on Dynamic Assessment see
references
Copyright Crowley, Agosto, Castle, Friedman, 36
Ouellette, Tancredi 2006
You should be asking
yourself…
Did I give this child the type of opportunities that
would enable him/her to demonstrate their true
language ability?
Did I give enough opportunities?
Are the tasks that I used appropriate for this
student’s sociolinguistic background?
Have I obtained information from his/her teachers
and caretakers?
Do I have information about him/her in a natural
setting?
Did I include holograms throughout the
evaluation?
Copyright Crowley, Agosto, Castle, Friedman, 37
Ouellette, Tancredi 2006
What to do Monday morning?
Meet with administrators to address the
relationship of assessments and
disproportionate referral of minority students
for special education.
Identify policy or local regulations that are
ineffective.
Gather research and volunteer for inservice.
Sleep soundly……
Copyright Crowley, Agosto, Castle, Friedman, 38
Ouellette, Tancredi 2006
References
Individuals with Disabilities Education Improvement Act of 2004 (IDEA 2004). 20 U.S.C. §
1400 et seq.
ADVANCE for speech-language pathologists and audiologists. www.advanceweb.com/speech
Crowley, C. & Ouellette, P. (2006). Introducing ABEL: Assessment based in evidence and
law. ADVANCE for speech-language pathologists and audiologists, 16(4), p. 10.
Crowley, C. & Ouellette, P. (2006) Asking the critical questions: The parent interview.
ADVANCE for speech-language pathologists and audiologists, 16(9), p. 10.
Crowley, C, Bielefeld, A., & Oullette, P. (2006). Putting tests to the test: Confidence intervals.
ADVANCE for speech-language pathologists and audiologists, 16(13), p. 10.
Crowley, C, Oullette, P. & Bielefeld, A., (2006). Putting tests to the test: Discriminant
accuracy. ADVANCE for speech-language pathologists and audiologists, 16(17), p. 9.
Crowley, C. & Ouellette, P. (2006). What IDEA 2004 says about assesssment. ADVANCE for
speech-language pathologists and audiologists, 16(23), p. 11.
Crowley, C. & Ouellette, P. (2006). Columnists respond to readers’ questions. ADVANCE for
speech-language pathologists and audiologists, 16(43), p. 17.
Gray, S., Plante, E., Vance, R. & Henrichsen, M. (1999). The diagnostic accuracy of four
vocabulary tests administered to preschool-age children. Language, Speech, and Hearing in
Schools, 30(2), pp. 196-206.
Heath, S.B. (1982). What no bedtime story means: Narrative skills at home and school.
Language in Society (Vol. II). New York: Cambridge. pp. 49-76.
Catts, H.W., Kamhi, A.G. (2005). Language and Reading Disabilities, 2nd Ed. Boston, MA:
Pearson Education, Inc. Fox, S. (2006). Are you ready, willing, and ABEL to update your
skills? ADVANCE for speech-language pathologists and audiologists, 16(4), p. 5.
Copyright Crowley, Agosto, Castle, Friedman, 39
Ouellette, Tancredi 2006
References (con’t)
Kratcoski, A.M. (1998). Guidelines for using portfolios in assessment and evaluation.
Language, Speech, and Hearing Services in Schools, 29(1), pp. 3-10.
Lidz, C. & Pena, E. (1996). Dynamic Assessment: The Model, its Relevance as a
Nonbiased Approach, and its Application to Latino American Preschool Children.
Language, Speech, and Hearing Services in Schools, 27(4), pp. 367-372.
McCauley, R.J. & Swisher, L. (1984). Psychometric review of language and articulation
tests for preschool children, Journal of Speech and Hearing Disorders, 49, pp. 34-42.
McCauley, R.J. & Swisher, L. (1984). Use and misuse of norm-referenced tests in
clinical assessment: a hypothetical case, Journal of Speech and Hearing Disorders, 49,
pp. 338-348.
Peña, E. & Quinn, R. (1997). Task familiarity: Effects on the test performance of Puerto
Rican and African American children, Language, Speech, and Hearing Services in
Schools, 28 (Oct. 1997), pp. 323-332.
Plante, E. & Vance R. (1994). Selection of Preschool Language Tests: A Data-Based
Approach, Language, Speech, and Hearing Services in Schools, 25, pp. 15-24.
Restrepo, M.A. (1998). Identifiers of Predominantly Spanish-Speaking Children with
Language Impairment, JSLHR, pp. 1398-1411.
Westby, C. (1990). Ethnographic Interviewing: Asking the right questions to the right
people in the right ways. Journal of Childhood Communication Disorders, 13(1), pp.
101-111.
Wolfram, W., Adger, C.T., Christian, D. (1999). Dialects in Schools and communities.
Mahwah, NJ: Lawrence Erlbaum.
Copyright Crowley, Agosto, Castle, Friedman, 40
Ouellette, Tancredi 2006