Tuesday, June 21, 2016

Specifications grading: Part 3 - Avoiding Student Confusion

This is the second in a series of posts about creating, implementing, and modifying a specifications grading system. You can find them all by clicking on the "specifications grading" label below. Or you can follow these links: Part 1, Part 2

The previous posts laid out the development of a specifications grading system. When I actually implemented the system in 2014-2015, I ran into two major problems: confusion and anxiety. I've made a number of changes to, hopefully, mitigate those problems. I'll update next year with my success (or lack thereof). There is a lot of good information in Linda Nilson's book Specifications Grading, but I found my specific problems were not detailed in the book, perhaps because I'd already taken her advice for dealing with the problems she covered.
Problem 1: Confusion

I devoted an entire page of my syllabus to explaining the specifications grading system. I described it in detail in class, multiple times. After the first graded assignment, I showed students how it fit into the system. At the end of each unit, I went over the grading system again and gave students a personal update on their grades.

I still lost people.

Honestly, at the end of the semester there were still some students (granted, only a couple) who had no idea how the grading system worked. There were students who took A-level final exams to try and "raise their grade" when they hadn't done any of the C, B, or A work earlier in the semester. When I tried to explain that this was impossible under a specifications grading system, they would ask if they could do it anyway. ("Why?" I asked. "To raise my grade," they replied. "Gee, Professor, why are banging your head against the wall?")

Very few students understood the grading system on the first day, but around 90% of them understood it after the first unit exam. Even they were annoyed by the complexity of the grading system because I had to keep explaining it in class for those who didn't understand.

I'm not going to show you the original grading system because it was too convoluted (I'm happy to share if you want me to send the link), but suffice to say I've greatly simplified the structure. I have two recommendations for battling confusion:

1. Find an analogy/example students can understand. I tried a variety of ways to explain and defend the grading system. Students are so used to the standard grading system (90%+ = an A, 80%-90% = a B, etc.) that they found the idea of a mastery system extremely confusing. Even after they understood the basics, students had trouble understanding why it didn't matter whether they got an 80% or a 100% on their multiple choice exam, just as long as they passed it. Through trial and error, I found a couple of analogies/examples that helped my students understand the grading system. YMMV.

One example that worked was the one I used in my first post: my high school chemistry class. That class consisted of self-paced units, each with associated lab work, homework, and an exam. You had to pass the exam with at least a C to move on to the next unit. Your final course grade was entirely dependent on how many exams/units you passed, not on your percentage grade on the exams. If you only did a couple units, you'd fail. If you did ten or more, you'd get an A. Since the first units covered very basic material and the later units covered more complex material, if a student got a D that meant they had mastered only the basic concepts, while an A reflected mastery of more advanced material. Although my BioAnth class is not self-paced, and every student has to complete some aspect of the three course units, they still found this example helpful in understanding the difference between a percentage-based grading system and a mastery-based grading system.

Visualization of the specifications grading system for Intro to BioAnth
The visual analogy that proved most effective was a (wonky-looking) tree or upside-down pyramid. To pass the course, all students must master the foundation/root work (D-level material), while higher grades required climbing higher (and covering more material).

2. Label graded activities carefully. Even if you have a complex class structure, by carefully choosing your labels for graded activities, you can make the grading system appear simpler. In my BioAnth class, I have a lot of different assessed activities: in-class writing, out-of-class writing, multiple-choice exams, essay exams, labs, in-class activities, worksheets, etc. Originally, I went through these activities and assigned each a grade level, then asked students to do only those activities that pertained to the grade they wished to obtain (and those of the grades below, of course). So D-level students had to do the D-level in-class writing, multiple-choice questions, labs, etc. C-level students did those, as well as all the C-level in-class writing, multiple-choice questions, labs, etc. It seemed relatively straight-forward to me, but it wasn't. Students got confused about what material they needed to cover and they lost track of deadlines.

I revised the system so that all the assessments that are required for a particular grade have the same label (even if they're not necessarily identical activities, and even if there are similar activities that aren't required.) For example, I assign weekly short essays. Originally, these were all called "comprehension checks" and I gave each essay a grade level; students who wanted a D in the class had to write the D-level comprehension checks. Students who wanted a C in the class needed to only write the D-level and C-level comprehension checks, etc. This was much too complicated. Now, I have divided the same weekly essays into two categories. The easier ones, the ones that cover basic concepts, I call "comprehension checks". All students who want a D or higher must pass them. The essays that require a more sophisticated understanding of the material I call "mastery checks", and they need only be completed by students who want an A.

Similarly, I divided my previous exams (which had multiple choice, fill-in-the-blank, and essay questions) into different sections, gave them different labels ("basic knowledge checks" for the multiple-choice questions vs. "midterm" for the fill-in-the-blank and applied skills questions), and scheduled them for different days.  Students no longer have to ask "what sections of the exam must I do?", they know they need to take all "midterms" and/or all "basic knowledge checks". As I laid out in the previous post, all the activities for a particular grade are given the same label on the syllabus, making it easier for students to understand what they need to do and when.

I'll talk about dealing with student anxiety in my next post.

Monday, June 6, 2016

Specifications Grading: Part 2 - Assessment

This is the second in a series of posts about creating, implementing, and modifying a specifications grading system. You can find them all by clicking on the "specifications grading" label below. Or you can follow these links: Part 1

In my last post, I laid out my class goals for Intro to Biological Anthropology and assigned a "grade level" to each. The next step is determining how to assess whether or not the students have achieved each goal. Again, I lean heavily on Linda Nilson's book and the work of Dee Fink on designing courses for significant learning.

Step one: matching assessments with goals

I literally just go through my learning goals and decide what type of assessment I will use for each one. I won't bore you with all the details, but here are some examples with the learning goal in black, the grade level in red, and the assessments in blue:

Foundational Knowledge: Students should be able to explain in their own words and give examples of the following key concepts:
  • Evolutionary theory D 
    • Multiple choice questions: ex., which of these is an example of natural selection?
    • Short essays: ex., define evolution and explain why it is not the same as natural selection 
    • In-class activities: Four forces of evolution; Pedigree analysis; Creating cladograms
Application: Students should be able to:
  • work "hands-on" with materials that tell us about human adaptation and evolution (fossils, biometric data, etc.) D
    • Lab write-ups and worksheets on fossils, primate skulls, and human osteology
    • Lab practicums
  • critically evaluate scientific arguments about fundamental human nature (based on race, gender, our hunter-gatherer past, etc.) B
    • Longer essay: ex., if your local newspaper published an opinion column that claimed males are better at math than females, how would you critique this from an anthropological perspective? Make sure you use actual data and examples from class or readings.
    •  In-class activities: Data on difference; Clines and population variation; What does it mean to be human?; Race is in the cards; Discussion: Race, housing, and systemic inequality; Discussion: sex, gender, and violence
    • Extra-credit opportunities: Six words project; #BlackLivesMatter reading and summary; campus speakers as available 

You get the idea. A few things to point out:
  • Not all assessments need to be exams or even graded. I count extra-credit opportunities or in-class assignments where the students are engaged with the topic and write a short reflection. Reading the reflections helps me revisit concepts that didn't get across the first time.
  • Higher grade-level goals have more complicated assessments, such as essays vs. multiple-choice questions. YMMV.
  • The more important the issue (from my perspective) the more ways it gets assessed in the class. Note how many things are listed under "critically evaluate scientific arguments about fundamental human nature", and I left off a bunch more. Even though only B students are asked to master that goal, all students are exposed to it again and again.
I had already taught this class several times before implementing the specifications grading system, so I had a whole list of activities, labs, and exams that already worked for me. Mapping them onto the learning goals, like above, helped me see where I had blanks and forced me to dump/modify some things that weren't serving a purpose.

Step two: develop a grading system (KISS)

Once I knew how I wanted to assess whether or not students had mastered a particular learning goal, I had to structure those assessments into a coherent grading system. Here's where I part ways a bit with Nilson's book. Something I wish I had known before I started: KISS

I'll dedicate another blog post to how badly structured my specifications grading system was the first time I tried it. Here's the new, improved structure:

To earn a D, a student must show mastery of the concepts in all "Comprehension Checks". These are short (one-paragraph) essays asking for definitions and examples -- or applications -- of basic class concepts. They are assigned every other week. 

To earn a C, a student must pass the requirements for a D, as well as answer 80% of questions correctly on all five "Basic Knowledge Checks". These are computer-generated multiple-choice exams, for which they get the questions ahead of time. (That doesn't help their grades as much as you might think.)

To earn a B, a student must pass the requirements for a D and a C, as well as achieve an 80% on three midterms. The midterms are also computer-generated exams for which they have the questions ahead of time, but they include essay questions and require skill application (like correctly calculating and interpreting Hardy-Weinberg, or pedigree analysis, or reading a cline map, etc.).

To earn an A, a student must pass the requirements for a D, C, and B, as well as show mastery of the concepts in all "Mastery Essays". Like Comprehension Checks, Mastery Essays are relatively short (2-3 paragraph) essays assigned every other week. Unlike Comprehension Checks, Mastery Essays move beyond defining or giving examples of class concepts and instead require students to grapple with more complex topics (for example: "In your class textbook, Jonathan Marks claims there is no such thing as a value-neutral primate taxonomy. What does he mean by that? Do you agree or not? Why?").

In addition, 20% of the grade for all students is based on three lab practicums which are graded on a simple percentage scale. That is, if they correctly identify 80% of the bones on the human osteology exam, they get 80% of the total points. Last year, I tried to use a specifications grading system for the lab, as well, but it was a logistic nightmare. I might revisit it as a possibility in the future.

My goal is to make the class impossible to pass unless a student has mastered the basic concepts. I want to encourage students, however, to aim for a B over a C, so the requirements for a B are not really that much greater than the requirements for a C. Before the specifications grading system, I tried to pitch exams so the average grade was about 80%. Under this system, I'd like more students to fall on the B side of that line than the C, assuming they're in that middle area of what once was the standard bell curve.

To get an A, though, requires a more complex level of engagement with the material. The "mastery essays" pull from a variety of sources and themes within the class, and they really do separate the best students from those who are struggling with the material.

Next time, I'll talk about troubleshooting the grading system, re-dos, and the grading load.