PARCC Readability, Part 3: Considering the Reader

In Part 1 of this series on readability and the Common Core aligned PARCC assessments, I looked at the readability of the PARCC reading passages themselves. In Part 2 the focus was on the task that students were asked to do based on those passages, i.e., the questions that they had to answer after reading those texts. This third part of the series looks at the other factor that influences readability: the student.

At the start, I believe that it is fair to note that Appendix A of the Common Core State Standards, which deals with text complexity, recognizes the student as a critical consideration in determining the readability of a text. Here is part of what that document says about matching students to texts:

“[H]arder texts may be appropriate for highly knowledgeable or skilled readers, who are often willing to put in the extra effort required to read harder texts that tell a story or contain complex information. Students who have a great deal of interest or motivation in the content are also likely to handle more complex texts.”

And Appendix A adds significantly, “Teachers employing their professional judgment, experience, and knowledge of their students and their subject are best situated to make such appraisals.”

So the importance of the reader in any measure of readability is universally acknowledged. Unfortunately, standardized tests cannot match texts to readers like the child’s teacher can; therefore, the ease or difficulty of any reading passage can never be fully ascertained. This is why good standardized tests are pilot tested to determine if some passages cause some readers comprehension difficulties not based on reading skills alone.

Constructivist reading research has identified five things about the reader that matter in the reader’s comprehension of text: reading skills, reader prior knowledge, reader cognitive development, reader culture, and reader purpose. Presumably, we would want a reading comprehension test to measure student reading skills and cognitive development. We also know the purpose of reading in the case of any standardized test is to answer questions after reading. But what of prior knowledge and culture?

We would hope that the test would not advantage or disadvantage a student because of differences in prior knowledge or culture.  We would hope, but years and years of standardized testing have shown that this is never the case. Over their many years of use, standardized tests have not been able to keep prior knowledge and culture neutral in assessing reading comprehension.

Prior knowledge is critical to reading comprehension. Essentially, the more you know about a topic before you read about that topic, the better you will be able to comprehend that material and the better you will be able to accommodate any new material you encounter during that reading. In an article titled, Individual Differences that Influence Reading Comprehension, Darcia Narvaez, a professor at the University of Minnesota, puts it this way, “If a person [who] has a great familiarity with a grocery store reads a text about a grocery store, the person will activate a grocery store script.” This script aids the reader in making sense of text. If the reader has little or no script to bring to the text, say if the script were about a trip to an outdoor market in Morocco, the comprehension could be more limited.

Cultural bias in standardized tests has been well-documented over the years. Here is just one example that I read about in a technical report from The Center for the Study of Reading (1981) that looked at response differences of Black students and white students.

“This item involved a passage about a visit of Captain Cook to a group of islands in the South Pacific. The critical section was,

 ‘he called them the Friendly Islands because of the character of their people. Today, the Tongans still provide visitors with a warm welcome.’

The test item asked for the meaning of the word character as it was used in the story. Most whites chose nature, the answer scored as correct. Blacks frequently chose style. This is a term used more in black than white communities, and it can be argued that in its colloquial sense style is more apt than nature as a synonym for character. It is apparent, at least, that style is not a wrong answer.”

With all the study that has been put into issues of cultural bias, we can expect that passages and questions from any reputable test developer will be vetted for this bias, but again, these issues can be subtle and might best be accounted for through extensive pilot testing.

Now let’s take a look at the sample passages in the PARCC that I have discussed in the previous two posts and see how they measure up on the background knowledge and cultural bias scale from one literacy specialist’s perspective.

Grade 3 – A Once in a Lifetime Experience

This is a pleasant and innocuous story typical of the Highlights for Children magazine from which it was taken. As I said in a previous post, the quantitative readability seems appropriate. The story involves a camping trip with two friends and a dad. Some background knowledge on camping, fishing and boating might advantage students over those who have no such experience, but I do not think the impact would be great. More problematic, for me, is that answering one of the questions requires knowledge of the word “jostle.” The word cannot be clearly defined through context and I would not deem the word as one that most third graders would know. While the text provides students with the definition of “bail” and “adrift”, students are left to their own devices on “jostle.”

Grade 4 – Just Like Home

As I stated in my first post in this series, Just Like Home, by every quantitative measure, is too difficult for use in a reading comprehension test for fourth grade. As far as cultural bias or giving advantage to some who might have greater background knowledge, the concern is slight. The story is an obvious attempt to include a multi-cultural perspective to the test, and that is a good thing. It takes place on a school playground and the concerns seem fairly universal. My favorite part of the story is when the protagonist, Priya, says that the only thing she likes about her new school is art, because she did not have art in her old school. I immediately thought that art had been cut in her previous school so that the kids could do more test prep, but now my bias is showing.

Grade 5 – Moon Over Manifest

This passage is taken from the 2011 Newbery award winning book by Clare Vanderpool. The passage is well written, but it has some characteristics that may make it challenging for some readers. The story is set in the 1930s and references to such things as pocket watches, satchels, storefronts and bustling townsfolk may prove problematic to some. Typical of Newberry award books, the author uses lots of rich figurative language that puts a further burden on the reader, fine for some fifth grade readers, but challenging for others with a less rich background knowledge. Finally, the passage assumes knowledge of what has gone before in the story, the main character is a veteran of hitching rides on trains with her father, which may throw off some fifth grade readers.

Grade 6 – Emancipation:  A Life Fable

This fable, written in the 1860s by the well-regarded proto-feminist author, Kate Chopin, seems an odd choice for a test passage. Because it was written 150 years ago, it is replete with word choices and sentence constructions that may be unfamiliar to 11- and 12-year-old readers. Here are two examples:

Here he grew, and throve in strength and beauty under the care of an invisible protecting hand. Hungering, food was ever at hand.

Back to his corner but not to rest, for the spell of the Unknown was over him, and again and again he goes to the open door, seeing each time more Light.

The fable is actually a good example of why quantitative readability measures like Lexiles are problematic. They cannot measure the impact of arcane language or parse the allegorical nature of a fable. This is an extremely challenging passage for a sixth grader, one that would be best used in the classroom with plenty of support from the teacher, certainly not in a testing situation.

Grade 7 – from The Count of Monte Cristo

The Count of Monte Cristo is, of course, the classic adventure novel written in the 1840s by Alexandre Dumas. Written in French, the text has been translated and updated many times over the years. The translation used by PARCC is identified as in the Public Domain, so I would assume the test passage is from an older translation. We know that older texts, which employ language patterns and structures that are unfamiliar to many children, provide a greater reading comprehension challenge than more contemporary texts, so once again the difficulty of this text cannot be accurately measured by a Lexile score.

The text is replete with vocabulary that seventh graders will find challenging and that will impact their comprehension. Words in the passage include countenance, lucidity, well-nigh, loathsome, delirious, ascertain, recurrent. While some of these words might be determined through context by a skilled reader, most cannot. The setting of the story, a dungeon in early twentieth century France during the Bourbon Restoration, also would provide readers with a unique challenge.

The Count of Monte Cristo is a wonderful adventure story and would make entertaining reading for a certain subset of skilled middle school readers. The advisability of using it as general reading in a reading comprehension test for seventh graders is highly questionable.

Grade 8 – Elephants Can Lend a Helping Trunk

This is a non-fiction passage describing an experiment conducted by a team of scientists to test the social cognition of elephants. The passage is clearly written, is cohesive, and for the most part does not present an extraordinary vocabulary challenge. An included photo should help students visualize the experiment.

A minor quibble with this passage is that it was clearly written by a person who uses United Kingdom English. So crows are called “rooks” and some sentence patterns are slightly different than American English.

The biggest concern with this passage is that the understanding of the entire passage depends on the understanding of the words “cognitive” and “cognition.” Indeed the first two questions deal with these two terms. The main body of the passage discusses the elephants’ skills at social cooperation, but the text never draws a clear line from the understanding of cooperation and an understanding of “cognition” as mental processes. I think this passage is appropriate for a high school class in psychology, but question it as a test passage for 8^th graders. I recently had a discussion with my college freshman class about the term “cognition” and some of its related terms cognate, recognize, metacognition, cogitate. I can report that the word cognitive was not in most of my college freshmen’s vocabulary.

Standardized tests by their very nature cannot be well matched to individual readers. Text matching takes a skilled and informed teacher with deep knowledge of her students, the reading to be done and the task to be completed. It is not fair to ask standardized tests to meet these criteria. I believe it is also not sound test science to choose as test passages those passages that contains vocabulary beyond most students in the grade, passages whose targeted audience was adults not children and passages that use archaic language and sentence structures.

The high readability of most of these passages, the unique challenges of the questions asked on the test and the failure of some of the passages to be considerate of the background knowledge and culture of many of the children who will be encountering these tests, guarantees that many children will struggle. Ultimately, the results of the PARCC tests will tell us more about the tests themselves than it tells us about the students taking the tests. One thing it will tell us, I believe, is that these tests are not useful for making any high stakes decisions about individual children, teachers, or schools.

The one thing I can guarantee as an outcome of these tests is that, overall, children living in areas of affluence will do considerably better than children living in areas of poverty. I can guarantee this because it is true of every standardized test ever given, and so it always shall be. If education reformers want to learn from standardized tests, this is the lesson to be learned. The real issue in education is inequity, not the ability of a seventh grader to parse The Count of Monte Cristo.