How Do We Assess Student Listening Skills?

By Monica Brady-Myerov

For two decades I wrote audio stories as a public radio reporter for Boston’s NPR news station WBUR, striving to capture the listener’s imagination and hold it.

My hope in those days was that after listening to one of my reports about a court case or drug addiction you would remember it. I hoped you would learn from experiencing the story.

What I’ve come to see as the founder and CEO of Listenwise, a listening skills company, is that there isn’t a reliable, universally acceptable, research-based method to test listening comprehension skills in the general population.

Maybe that’s because there also isn’t a way to “level” listening. We have a lot of teachers ask us what Lexile level our public radio stories are. But there are many reasons why a straight Lexile level is not the way to go for an audio story.

Interestingly, research shows students can listen two to three grade levels higher than they can read. So we did a lot of research into this and came up with a new way to “level” listening.

It’s different from what anyone else has ever done, and I think it’s going to set a new standard for auditory learning in the classroom, and a new way to talk about tracking and building listening.

At Listenwise, we label listening complexity as low, medium and high. The levels do not relate to the content of the story, but to the complexity of the vocabulary, sentence structure and language used to tell the audio story.

After establishing our listening levels, we were ready to tackle a listening quiz.

The Common Core: Listening for learning

If you google “listening skills,” you will see 7 million results about the importance of listening and how to become a better listener. But the links are primarily targeted to communication skills in business. I agree that listening in business, especially in sales, is critical.  But when more than 23 research studies show better listening leads to better learning, more attention needs to be focused on listening in middle and high school.

Listening is a critically important skill to improve. And to improve it, you must be able to practice, assess and track progress.

The Common Core has elevated Speaking and Listening to an anchor standard. Some Common Core assessments are testing listening as part of the English Language Arts portion. In states such as Florida, which is not using the Common Core standards, there is still a significant portion of the ELA state exam in the middle grades based on listening. In the CAASPP, the state tests in California, the listening comprehension questions comprise 15-18% of the ELA exam.

High stakes listening

In most ELA state assessments, a student listens to a clip of nonfiction text (less than two minutes long and without the transcript), and answers five to seven questions about it. They can listen as many times as they want.

And though such tests are included in ELA state assessments, there’s no tool to help middle and high school students practice the discrete skill of listening. Yet research shows it is a skill that can be improved with tracking and practice.

Listening tests you find on the internet focus on helping people who are learning to speak English practice their listening skills. The tests are not tailored to native speakers. And they are not meant for the K12 classroom. So we saw the need and built a multiple choice formative listening assessment.

How our assessment works

We partnered with a cognitive scientist at the Massachusetts General Hospital to study and learn more about what makes up the key elements of listening. As an expert in reading comprehension, Dr. Tiffany Hogan has done foundational research on how reading and listening are closely tied. Her research has served as the basis of our listening assessment.

Our curriculum team established a listening skills matrix to identify the key elements of listening that we wanted to assess and how they align to the Common Core State Standards in Listening and Speaking, as well as reading comprehension. (Currently the Smarter Balanced Assessment assesses listening as part of the ELA portion of the test; listening is optional in PARCC.)

We identified how we could test strategic listening, precise listening, and critical listening as discrete skills. When we had the matrix ready, we worked with an expert in listening who wrote a textbook on how to improve listening. It  was a paper textbook on listening with no audio.

When we were done, we’d developed a set of effective listening comprehension strategies for teachers to improve student listening skills. Because, after all, it’s unfair to test listening skills if they were never taught in the first place.

A mysterious problem arose

Then we faced a challenge. No one had experience writing test questions to assess listening.  The closest consultants we could find were item writers in reading. So we hired experienced reading item writers. We selected audio stories from our platform that were less than 5 minutes long.  We shared the audio files with the writers and then waited.

In the first round of alpha testing, our team and some students took the quizzes. We found that on most of the key elements, the writers’ questions were accurate. But in the question about the “main point” of the story, there was always something a little off.

The main point question posed by the test writers wasn’t wrong exactly. It just wasn’t always what I or our editor thought was the main point of the story. Why was that?

We had a deep dive call with our team leader of the item writers and walked through the process of how the writers created the quiz. Here’s what we learned: They received the assignment, they were given the audio file and the transcript, and they started writing items, referring continually to the transcript.

“At what point in this process do they listen to the story?” I asked.

The startling answer: They weren’t listening to the story! They were reading the transcript and basing the questions on the text, just as they would a reading test.

But this is not a reading test, it’s a listening test. And uncovering this flaw in the process highlights the very reason why a listening test is so important.

When you hear a human voice tell a story, giving emphasis to some words and not others, and interviewing people who bring their own tone and inflection, you hear (and understand) something slightly different than what you would read.

Once the writers adjusted their process and listened to each audio story before creating item questions, everything began to fall into place.

What teachers say about our assessment

We’ve just completed beta testing with teachers and students around the country, and the process has been revealing to teachers. They found that students don’t listen as well as teachers might have guessed. And detecting inference when listening was hard for many students.

Editor’s note:
This promotional video explains how Listenwise works.
It’s not an endorsement by MiddleWeb.

Bryan Sheckman, a social studies teacher who beta tested at Lowell High School, said: “My students love listening as an alternative way to learn. I have embedded listening strategies into my teaching to focus on inferencing and main idea, because their previous state test scores showed weakness with inferencing. And now that listening is tested as part of the Common Core, I’m putting a stronger focus on listening in my teaching.”

We’ve also found that putting a label on skill strands, such as summarizing and citing evidence, allows teachers to talk about performance in a specific area.

Another teacher, Heather Ryan from Ormand Beach, said: “Students figure out how they learn best through immediate feedback on their quiz answers, and they can internalize where their strengths and weaknesses lie. After trying out 3 quizzes with my class, I can already see improvement on certain comprehension skills.”

It’s been great to hear teachers describe how students have responded. For example, it’s clear that students like reading along while they listen, and they do find comprehension easier than when they’re just listening.

They are discovering what we already know. Listening is different from reading. And that’s why authentic listening assessments are critical to building better listening skills and thus better learners.

Monica Brady-Myerov is a long time public radio reporter, most recently with WBUR. She left reporting to start Listenwise and help teachers use public radio in the classroom. Listenwise, with Common Core lesson plans built around story topics, is free for teachers (with limited features) and has premium features for a school or district subscription.