Readability of Sample SBAC Passages

In three earlier posts, I took at look at the readability of sample passages for the PARCC assessments which are being used to measure student progress on the Common Core State Standards (CCSS) in some states. You can find those posts here, here and here. As I stated in those posts the concept of readability is complicated and includes quantitative measures like readability formulas, task considerations and qualitative considerations including assessing how the text will match up with the reader.

In this post, I look at the same measures as they relate to the Smarter Balanced Assessment Consortium (SBAC) tests that are being used in other states. I looked at one reading passage on each grade level of the SBAC from grades 3 through 8. I found significant differences in the readability of these passages from what I found in the PARCC tests.

First, for the quantitative measure of the SBAC passages. I used several different readability formulas. Both the SBAC and PARCC tests use Lexile measures to determine readability. I added other commonly used measures of readability as a check against the Lexile levels. As I cautioned in previous posts, quantitative measures of readability are often imprecise, so I used several measures to see if I could get some sort of consensus on the passages.

In this table the Lexile score and the range considered appropriate for grade 3 is provided. Flesch-Kincaid and Fry measures are stated in terms of appropriate grade level. The Flesch Reading Ease score attempts to state the relative ease of reading a passage. A score of 90-100 should be relatively easy to read for an average 11-year-old. A score of 60-70 should be easily understood by a 13-year old.

Quantitative Readability

3rd Grade Passage - A Few New Neighbors

• Lexile Level -                                                      510 ((3rd Grade range is 520 - 820)
• Flesch-Kincaid Readability Measure (FK)      1.8 Grade Level
• Fry Readability Graph (Fry)                             2.5
• Raygor Readability Graph (RR)                       2.5
• Flesch Reading Ease (FRE)                               94.9

Summary - This passage should be relatively easy to read for an average third grader.

4th Grade Passage - Coyote Tries to Steal Honey

• Lexile       900 (4th Grade range is 740 - 940)
• FK            4.9
• Fry           5.2
• RR           4.8
• FRE         93.2

Summary - The consensus of the measures indicate that this passage falls in the upper range of readability for a fourth grader. A challenging, but not overly challenging passage by these measures.

5th Grade Passage - A Cure for Carlotta

• Lexile      660 (5th grade range is 830 - 1010)
• FK           5.8
• Fry          6.5
• RR          4.5
• FRE        76.8

Summary - The Lexile score seems out of step with the other measures on this passage. I will look more closely at the passage below.

6th Grade Passage - Fishy Weather Conditions

• Lexile      1040 (6th grade range is 925 - 1070)
• FK            7.5
• Fry            8.1
• RR            4.5
• FRE          70

Summary - The Raygor measure is out of step with all other measures, which provide a consensus that this is a challenging text for 6th graders. Again we will look at qualitative aspects of the passage below.)

7th Grade Passage - Life on the Food Chain

• Lexile      900 (7th grade range is 970 - 1110)
• FK            6.9
• Fry            7.1
• RR            4.5
• FRE          68.3

Summary - Once again the Raygor measure is out of step with the others. The consensus is that this passage should be very readable for the average 7th grader.

8th Grade Passage - Ansel Adams, Painter with Light

• Lexile     1090 (8th grade range is 1010 - 1185)
• FK           8.3
• Fry          8.8
• RR          5.3
• FRE        65.8

Summary - Once again the Raygor is anomalous, but the consensus here would be that the passage is appropriately challenging for average 8th grade readers.

Task Analysis

The task of a reader taking a standardized test is, of course, to answer questions. I looked at all the questions attached to these passages to determine what tasks were being required of students. For my analysis, I used the question categorization scheme developed by Dr. Taffy Raphael, Question Answer Relationships (QARs). QARs divide questions by the type of work the reader must do to find the answer to a question. Questions are categorized as follows.

• Right There; These are literal level questions whose answers can be pointed to directly in the text
• Think and Search: These are comprehension level questions like main idea questions that require the reader to put together an answer from pieces of information throughout the reading.
• Author and You: These are inferential questions, requiring the reader to use text evidence and his/her own background knowledge to answer the question.
• On Your Own: These are questions that are unrelated to the reading of the text. These types of questions are rarely seen on standardized tests.

I looked at 46 questions attached to the passages described above. Here is the breakdown as described by QARs.

• Right There:                   1
• Thank and Search:         17
• Author and You             28
• On Your Own                0

As would be expected from a test tied to the CCSS, a number of questions asked students to cite evidence for their answers. In the PARCC test this accounted for almost 50% of the questions. On the SBAC this percentage was closer to 30%. Every grade level was asked a question requiring determining the meaning of a word from context. This is also aligned with skills emphasized n the CCSS. Every passage also included questions aimed at the understanding of key ideas in the text and at an overall understanding of the text. While some questions were aimed at text analysis, the balance on the SBAC appeared to me to be more in keeping with a focus on a general comprehension of the text than were the PARCC samples I looked at, which were more focused on passage analysis.

Qualitative Analysis

Since quantitative measures of reading difficulty are notably unreliable, a third factor we must look at is qualitative, i.e. how we think the text will match up with the readers who will be reading it. Ideally a test passage will not disadvantage students because of different background knowledge or culture. In reality we know that standardized tests have difficulty doing this because they are targeted at such a broad audience. Here I look at each of these passages to determine as best I can how they will match with the target readers.

3rd Grade Passage - A Few New Neighbors

A straightforward and pleasant story that follows regular narrative structure. Vocabulary appears very appropriate for a third grade reader.

4th Grade Passage - Coyote Tries to Steal Honey

This passage is a folk tale that also follows a regular narrative structure. The trickster tale should be familiar to most fourth grade readers because so many folk tales are focused on a trickster, whether it is a rabbit, a raven or a coyote. Vocabulary in this tale appears to be well within the wheelhouse of most fourth grade readers. The use of figurative language may cause some readers minor issues in comprehension, but this passage appears to be appropriate for a fourth grade reader.

5th Grade Passage - A Cure for Carlotta

This story of a young boy's immigration to America on a ship from Italy is typical of many other stories aimed at elementary age students studying the story of immigration. The structure is a straightforward narrative with more descriptive detail than the passages for the younger students. Vocabulary load does not appear overwhelming for most fifth grade readers.

6th Grade Passage - Fishy Weather Conditions

This nonfiction passage is informative and entertaining. It explains the unusual phenomenon of fish falling from the sky in some areas of the world. The passage has a fairly high readability level for a sixth grade passage, likely due in part to the introduction of unfamialr vocabulary. Words like "dissipate" and "phenomenon" and "adaptation" might cause readers some challenges, but "dissipate" is directly defined in the passage and skilled readers can probably deduce the other meanings from context. Some figurative language like "connect the dots" may challenge some students. All in all a challenging passage that will cause some grade 6 readers difficulty.

7th Grade Passage - Life on the Food Chain

This nonfiction passage provides a straightforward explanation of the food chain. The text is organized in such a way that it should be easy for 7th grade readers to follow. The vocabulary load is heavy, but almost all terms are clearly explained right in the text. Sentence structure is not overly complex. A fair passage to assess 7th grade readers.

8th Grade Passage - Ansel Adams, Painter with Light

This biographical piece is written in a narrative format, telling the story of how Ansel Adams came to be a great photographer who chronicled the beauty of the American West. The passage contains a good deal of fairly sophisticated sentence structures that may cause some readers difficulty, but in general the account is highly readable. There are few concerns with the level of vocabulary for an eighth grade reader. I think the passage is appropriate for an 8th grade assessment.

Conclusions - 

1. Unlike the passages I reviewed for the PARCC test, I think the passages I examined from the SBAC test are fair representations of what children in those grades can and should be able to read.
2. The questions asked about these passages seemed to me to be a good mix of comprehension based questions and analysis based questions. In general the questions seemed appropriate to the text.
3. The passages chosen for the assessment all appeared to me to be straightforward enough that most students could follow them. There were no passages using archaic language or structures, no stories written long ago. Vocabulary was generally reasonable and often defined in the context of the passage.

Cautions:

1. I sampled only one passage for each grade level, so other passages may have problems I did not see here. Only by actually having large numbers of students taking the test will we be able to tell if the test meets industry and common sense standards of validity and reliability. 
2. Just because I have judged this test to be a reasonable test does not mean that I think this test, or any standardized test, can be used for making high stakes judgements about children, teachers or schools. The failure of standardized tests to be helpful in these areas has been well established. True understanding of individual readers' strengths and weaknesses is best done by professional educators working with children over time.

Test passages that offer most students at grade level the opportunity to demonstrate their actual reading ability can give teachers data that can help to inform instruction. In this cursory look at the SBAC test, it looks like these tests could meet that standard. Time will tell.