Question Level Analysis – The Good, the Bad and the Ugly

As teachers we want to make the biggest impact on our students. However we also know that just because we’ve taught something doesn’t necessarily mean that it’s been learnt and retained by our students!

For the last four years I’ve been using Question Level Analysis (QLA) of tests and assessments to try to better understand what my students have really learnt. This has been hugely powerful; it’s allowed me to see which topics I need to re-teach and which areas students need more practice on. This is even more important since I was awful at predicting how my students would perform; my predictions were wrong 59% of the time and I tended to over-predict their performance. After targeted re-teaching and further practice followed by re-testing, I’ve found that my student’s performance has increased by 20-25% on average, which is an equivalent increase by 1-2 GCSE grades.

It’s also had a powerful effect on my students. For instance a student that got a low overall number of marks (10 out of 60) gained more confidence when she saw that most of those marks had come from just two different topic areas. She could see that she knew those really well and that she could answer the questions on them. She was no longer disheartened by her low overall score. This motivated her to go and work on some of the other topics she hadn’t performed so well on.

However, like anything in education, to get good results, QLA needs to be used in the right way.

Claw hammer

Source: Wikipedia.

For instance a hammer is a great tool for hammering nails into wood. However would I get the same result if I tried to hammer a nail with the claw part of a hammer? Probably not! Just because something is a good tool for a job doesn’t mean that it can’t be used in the wrong way, and so give disappointing results. The same applies to QLA.

In this article I’m going to highlight what you can do to get the most from QLA (the good) by looking at what not to do (the bad and the ugly).

The Ugly…

An example of a really bad Question Level Analysis can be found in this blog article by Jasper Green. I’ve reproduced the image below from his article.

In his analysis Jasper says ‘Take this exam question, the topic is electrolysis. This student scored 1/3. Analysis of the paper by topic would suggest that this student needs to go away and learn electrolysis. But do they? If we look more closely at the individual elements of this question we can see that there are actually some other, much more fundamental aspects of chemistry that this student does not understand.”

The first mistake that Jasper has made here is to use the score for the whole question. Now this can only be done if all parts of the question are testing the same thing. If they aren’t then your analysis is going to be misleading.

The second mistake here is that Jasper has confused the context of the question with what it is actually testing. Although the context is electrolysis the question is only testing a very small part of a students knowledge of electrolysis. In fact part (b)(i) is not testing electrolysis at all, and (b)(ii) is testing if the student knows a use of a product of electrolysis, not electrolysis per se. Only part (a) is directly testing electrolysis.

So we need to look at each question item rather than the whole question, and what each item is really testing, not the overall context of the question.

So what are these three items testing? I’d argue that part (a) is testing if the student can predict the products of electrolysis, (b)(i) is testing whether the student can write the formula of simple covalent compounds, and (b)(ii) is testing if the student knows a use of a product of electrolysis. We also need to acknowledge that (b)(ii) is a multiple choice item and so getting this correct doesn’t necessarily mean that the same student would be able to give a correct written response to a similar question.

If we mock up what the QLA would look like in a spreadsheet for Jasper’s analysis versus mine, we get a very different view (see side image).

Here the top image is the QLA Jasper would have seen given his analysis; not very useful. The bottom is the QLA we’d see by looking at each question item and what each item is actually testing, not the context.

This second QLA is really useful as now we can say that the student needs to practice predicting the products of electrolysis and writing the formula of simple covalent compounds.

Jasper makes a third mistake when analysing this students responses: making invalid conclusions.

In his analysis of the students response to part (a) Jasper writes “they don’t know that elements are always conserved in chemical reactions”.

Maybe, and maybe not. The student could have just guessed carbon dioxide. I’ve actually heard one student say in class “if in doubt, just put carbon dioxide as it comes up so much”. But we can say that the student needs to practice predicting the products of electrolysis, and if they don’t understand that elements are conserved during chemical reactions then working on this will help that too.

Now lets look at Jasper’s analysis of part (b)(i): “The student states that the formula of a molecule of chlorine is Cl instead of Cl2. The student clearly does not understand the concept of diatomic molecules. Simply reviewing the paper in class and getting students to make corrections will only bring about progress if that exact question appears again. A much more effective approach would be to review diatomic molecules and covalent bonding”

Again maybe the student doesn’t understand diatomic molecules. But maybe not. I don’t think you can say that from this one question item. Usually students who write Cl instead of Cl2 can draw a correct dot-and-cross diagram of a chlorine molecule, and students won’t usually think of this when they are writing out the formula. The student could understand diatomic molecules but have simply forgotten to write Cl2. So here all that’s needed is to check if they can accurately draw a correct dot-and-cross diagram of a chlorine molecule and be reminded that most non-metallic elements exist as diatomic molecules (Cl2, H2, N2 etc.).

Whenever we are thinking about why a student may have got an item wrong we have to be careful that we don’t read too much into it. One item doesn’t tell us all about a students understanding of a particular topic, and we could easily come up with the wrong reason for why students get questions wrong. The QLA gives us a starting point; there will usually be a range of reasons why a student could have got a question item wrong. If we bear this in mind we will be able to design tasks that enable students to practice all the skills that are related to the question item and that can give us further feedback about student’s knowledge and skills.

The bad…

To illustrate the fourth mistake we can make with QLA, look at the following GCSE Mathematics question, taken from OCR. What is it testing?

Source: OCR J560-06 H SAM

I’d argue that this question is testing two things: forming equations from word descriptions and substituting one equation into another to solve for an unknown.

However if a student answers it completely incorrectly (or leaves it blank) we don’t know which part was the problem. Is it that they can’t write equations from word descriptions? Can they substitute one equation into another to solve for an unknown?

The problem is this question in not diagnostic. And in fact most past exam questions are not. So using assessments only composed of past exam questions is flawed. They are designed to separate students, not diagnose difficulties.

That doesn’t mean that we shouldn’t use past exam questions in our assessments. We need to know how well students can answers these, as well as getting them used to the style. But if they are all we use, then the conclusions we can draw about what our students know and can do are limited.

So what can we do to improve our assessments and make them more diagnostic? A better way is to complement any past exam questions with questions that test each skill or item of knowledge individually. That way we can more accurately determine what it is that is causing our students difficulty.

So if we were using the above past exam question in an assessment, we could also include the following questions in different parts of the assessment:

  • Marty and George complete some homework. Marty takes 15 minutes longer to complete his homework than George. Write this as an equation.
  • Find the values of M, G and B if M = G +15, B = 3G and M + B + G = 200.

Each of these questions tests the same knowledge and skills as the above exam question, but because we test each part individually we can quickly diagnose what is causing our students difficulty.

The easiest way to do this is to have assessments in two parts. One part is composed of past exam questions whilst the other part is composed of diagnostic questions that test the same knowledge and skills as the exam questions but individually. These parts can be given to students either at the same time (effectively as one assessment) or at different times (one or a few lessons apart).

The good…

We’ve seen the common mistakes it’s easy to make with QLA. In order to get the most out of QLA we need to:

  • Analyse the marks for individual question items, not overall questions (if they have multiple parts).
  • Be specific with what the question item is actually assessing. This may not be the context of the question.
  • View the QLA as the starting point; there will usually be a range of reasons why a student could have got a question item wrong and it is very easy to jump to the wrong conclusion. If we bear this in mind we will be able to design tasks that enable students to practice all the skills that are related to the question item and that can give us further feedback about what our students know and can do.
  • Not using assessments composed of only past exam questions. Include diagnostic questions that test the same knowledge and skills as the exam questions but individually.
0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *