conversations with Tod Massa about assessment

For those interested in this kind of thing, I storify-ed some recent twitter conversations I had with Tod Massa about the past week’s articles in the Chronicle of Higher Ed regarding assessment.

Topics included: measurement’s reduction of teaching and learning into countable, thing-like objects; the role of understanding as object of measure and goal of measurement; why knowledge is not a heap of stuff, however valuable; the risks of indirect measurement; how measurement, when properly pursued, can help us perceive changes over time.

DM

why i hate writing learning objectives*

[*even when they’re labeled “course goals”]

I’ve been puzzling over the course goals etc etc for a new grad seminar in pedagogy I’m teaching this spring, and I think I finally pinpointed the single most frustrating aspect of the language of assessment for me, especially when it’s used to guide, direct, or evaluate instruction. It’s the reversal of priorities it seems to entail, when assessment drives pedagogical decision-making instead of the other way around.

Assessment, if it’s viewed as something that manages or directs pedagogy, threatens to take faculty away from all the stuff that we love and value in teaching (e.g., literature, disciplinary research, students, discussions, interactions etc) towards stuff that we may never love, or only barely value (e.g., quantification, social science notions of data and evidence, standardized teaching methodologies, bureaucratic protocols of compliance).

Yet even with these caveats, I still believe that these kinds of assessment exercises have the potential to improve our instruction, so long as they’re conceived as another form of feedback for faculty to use in the creation and revision of our courses. And I think that any course about pedagogy nowadays needs to introduce future teachers to the complex relation of assessment to one’s classroom practice.

Some of the best, most lucid discussions of these issues can be found in Erickson et al.’s book, Teaching First-Year College Students, which is designed to help instructors of first-year students understand the sheer difficulty and significance of this transition for students. But the book is comprehensive enough to help new teachers at any level understand the challenges of teaching and learning in contemporary universities.

So here’s the paragraph I was using to think about my own learning objectives/course goals, in Erickson, et al., p. 71:

Erickson p.71

The process of drawing up these course goals begins by moving the focus away from the person teaching the course to the students taking the course. In other words, we move from “the course will do X” to “students will be able to do Y”) This is a difficult but useful shift in perspective that I think most teachers would endorse.

What is truly counter-intuitive is the major shift identified in the quote: “indicate the behavior expected, not the state of mind students will be in” [emphasis mine]. In other words, what outward behavior or activities manifested by students would provide visible, or even measurable, evidence that students are indeed “knowing, understanding, or thinking” the content of your course? What kinds of evidence can you provide that would corroborate your intuition that student A knew, understand, or thought better than student B?

I believe that experienced teachers intuitively regard “student thinking” as something that they are able to engage with, understand, assess, or try to improve, even if our intuitions and experience can be shown to be fallible.

Redirecting teachers’ attention strictly to student behavior, however, takes us away from our perceptions of students’ thinking, and often forces our attention on the lowest-level tasks and students’ demonstrated acts of compliance, which are of course the easiest parts of student activity to measure. The extent to which we demand that students “know, understand, or think” seem to vanish from this minimalist depiction of learning. And higher ed teachers are particularly baffled by this kind of goal displacement, when discussions of “critical thinking” or “higher-level learning” ignore disciplinary “ways of thinking” that remain tacit or opaque to outsiders.

Unless really ingenious methods of indirect observations are put into effect, the minimal, behaviorist picture of learning is where most of assessments of the learner and learning remain. They essentially inform us of the number of students attending classes and the number of hours they filled seats and drew upon “resources,” meaning instructor time and possibly attention. In some sense, the “competency movement” represents the instructional model that this kind of assessment and its advocates would move towards, but there are real questions about whether it can be done credibly enough to compete with more traditional educational approaches. But the biggest difficulty for all these externally-focused programs of assessment is that they are uninterested in the quality of those interactions or learning that would define an experience as “education” in our usual sense of the term. There is nothing transformative, or potentially transformative, in these kinds of experiences.

The most infuriating aspect of this situation, however, is when this behaviorist language of assessment, once it has rendered most higher-level work invisible, demands that something called “critical thinking” or “upper-level learning” be taught and assessed using the methods that are least suited to generating or observing them. In this scenario, true evaluations of success or failure are essentially irrelevant to the system getting built, because it is outside the control of the student or teacher to alter.

Having said all this, I agree with Erickson that an integration of pedagogy with assessment (via strategies like provisional, instructor-written course goals) remains a worthwhile activity, because it helps clarify to ourselves and our students what we’re attempting to do. In other words, this integration of pedagogy of assessment should be done to the extent that it improves our teaching or our students’ experiences of learning. And anything beyond that feels like a displacement of our genuine goals and values regarding teaching.

DM

incremental learning (on all sides)

Now that the term is over, I’m doing my usual review of the previous semester, and a question came up in my mind: why is it so much easier to improve your teaching incrementally rather than all at once? Why do the attempted, full-scale reinventions fall flat, when longer-term, more piecemeal improvements seem to work better initially and have more lasting results?

There are trade-offs both ways. Doing it all at once gives you the opportunity to start with a clear conception and see it all the way through.  Tweaking is less risky, because you’re usually beginning with something that you’ve inherited or established that feels at least functional, but often you feel like you could be doing things without really understanding the rationale.  The initial impetus has gone away.  But I can say that my most successful teaching has always been in the long-running courses that I’ve had the opportunities to rework year after year.

This semester in my Swift and Literary studies course, which I’ve previously blogged about, I had some small assignments that seemed to help my students in significant ways.  I created these largely because I was concerned about the reading skills of students coming into this course, the gateway for the English major. Here they are:

1. “Representative Passage” assignment for Gulliver’s Travels. I developed this because I felt that students were reading so much criticism and theory that they tended to focus on very obvious passages or episodes from GT for their final assignments.  This assignment was based on some exercises I found in Kathleen Blake Yancey’s Teaching Literature as Reflective Practice and Blau’s Literature Workshop (discussed earlier on this blog).

The idea was to get students (most of whom had never read Swift or GT or studied much prose besides short stories) to record some key information for each part of GT, then select a passage related to some question for that week’s discussion. Each week, students individually filled out a worksheet describing their choice of passages, then discussed their selections in groups, with the groups choosing one passage then reporting their choices out to the class as a whole. Afterwards, I’d look at the worksheets and recorded discussions, give them a check, check minus, or check plus credit, and return them. I repeated this exercise four times, once for each part of GT. At the end of term, a number of students mentioned how helpful this exercise was for them to hear about other students’ thinking about the selections. And the final papers did feature a wider than usual range of GT passages than I’d seen before.

2. In-class essays. I originally assigned short response essays on topics in critical theory, but I eventually realized that they wrote better timed in-class essays than response essays on these topics.  Then I started collecting their questions on the theorists to create the in-class exam, adapting them as necessary but still leaving them options so they could choose their questions.  Finally, rather than doing these simply as open book or open notes, I allowed students to create a single typed or printed page of notes to bring to class, on the condition that these were handed in along with the completed in-class essays. These note sheets helped me assess students’ understanding and synthesis of the material on the essay, and like their essays, when handed back with feedback, these sheets became another source of ideas for their final research projects. They repeated this cycle twice, just before embarking on the final projects. I think this kind of cycle (questions, note-sheet, in-class essays, feedback) is a good way to teach theoretical topics that ordinarily only the most self-assured students feel comfortable enough to discuss.

What I’ve learned from this is that incremental, recursive cycles during the semester really help them develop the confidence to learn and discuss what they’re learning, but that this is in effect my cycle, too, as I teach the course from term to term.

DM

blau’s literature workshop: interpretive dependence

Apart from his interesting views about the theory/practice split in literature teaching,  Blau has some good insights into the “interpretive dependence” we help instill and reinforce in literature students when we teach in the conventionally authoritative ways.  (This seems equally true in both lecture- and seminar-style courses, incidentally).

In the conventional scenario, the literature professor usually acts as the authoritative source of knowledge. This kind of professor anticipates student confusions with carefully synthesized prefatory lectures; confidently answers questions (because he* [*since this version of authority is heavily gendered] has  answered them a thousand times before); and spontaneously produces the summary remarks that tie together in-class conversations with the authoritative views of experts.  And hey presto! you’re done.

This is the kind of pedagogical model many of us remember from our own education and would love to emulate, because who wouldn’t want to become this kind of authority?  And I admit that even though I share Blau’s skepticism about this kind of teaching, my bad days with things like groupwork or student presentations sometimes make me wish I had a more traditional classroom.

So what is wrong with this model of teaching?  What we should notice is how the satisfactions and learning end up belonging to the figure of authority.  Its successes are not successes of learning, but of teaching.

Blau points out that this kind of teacher-as-authority has cleared away the precious source of any inquiry, which is readerly confusion.  Though Blau doesn’t quite say it this way, the difficulties we encounter in literary works are not obstacles to understanding, but the best way in to deeper, more refined, better integrated understandings of the works we study, through independent inquiry.  His heuristic principle is that “the only texts worth reading are the texts you don’t understand. Because if you understand a text as soon as you read it, you must have understood it before you read it, so you don’t have to bother reading it in the first place” (24).

From the pedagogical perspective, students need to know that it is the texts they don’t immediately understand that may be especially worth reading, because “the process of figuring it out will constitute a process of advancing or deepening your vision.”  This is his way of arguing the value of unfamiliar (yet canonical) texts to the hostile, skeptical or indifferent student who will blurt out, like one of the students he observes, that “this play sucks,” when the class is reading Shakespeare. (Note, however, that this kind of student resistance happens at least as often, if not more, when reading more recently canonized authors.)

From the disciplinary perspective, though, Blau seems to have identified an important aspect of professional literary study, which we might call the literary counterpart of Wineburg‘s “historical thinking“: “interpretive thinking.”  This emerges from his account of the differences between professional and novice readers of a Thoreau passage, where the professionals are actually eager to find the areas of difficulty or confusion that leave novices frustrated, stymied, or depressed at their own insufficiency as readers (28-32).  And the predictable response of such students is to desire, or demand, that an authoritative teacher clear up those problems that could serve as the beginning of their own learning.

Blau’s alternative is the “literature workshop,” which models to students a collaborative, self-revising, process of discussion and synthesis that will support its members as they work their way towards those more refined understandings.  Students should watch teachers engaged in their own struggles to understand questions and devise solutions, and realize how much individual work and outside feedback is necessary to arrive at a finished piece of scholarly interpretation. Though Blau treats the literature workshop as a way of training students to engage in individual inquiry, it seems to me equally important to persuade students to commit fully to a collective process of inquiry and synthesis of multiple perspectives that will benefit them all.

Blau observes that this process-oriented, workshop model is commonplace in first-year and creative writing pedagogies, but much less common among literature instructors.  It would be an interesting question to investigate: in what ways does the study of literature reinforce the traditional dyad of interpretive authority and dependence? And why might this kind of interpretive authority and mode of interaction be seen as a problem, even by some literature scholars, at this point in time?

DM

blau’s literature workshop: some premises

I’ve been thinking about Sheridan Blau’s The Literature Workshop for a while now, because I’m slated to create and teach an advanced graduate course in pedagogy in the spring, and I’ve been looking for approaches that would give more experienced teachers some new concepts and practices to use when they return to the classroom.  The emphasis on the “advanced” pedagogy comes from my experience that most texts on higher ed teaching seem designed for beginning instructors. Texts like Jim Lang’s On Class or McKeachie’s Teaching Tips do this job very well, but they have almost nothing to say to the teacher who already knows how to face a class on the first day of the semester, plan a syllabus, or grade a paper, but would still like to improve her teaching in specific ways.

In other words, once we get past our existential fears of facing students and acting as authority-figures, how can we identify and work on the things that still trouble our work in and out of the classroom?  Unlike the fears everyone experiences in their first years of teaching, these problems come at us in very specific ways throughout a semester, and will continue to visit us semester after semester, if we cannot find good solutions.

Compared to the worries of beginning teachers, the problems faced by experienced teachers are both very concrete and frustratingly difficult to overcome, since they often involve working between contradictory pedagogical or professional principles that need to be maintained simultaneously. How to provide better feedback, without losing oneself to a stack of papers every other week?  How to lead discussion more effectively, given the unpredictable mix of passivity and self-display your students might demonstrate?  How to design better assignments, if you feel that your current assignments are only reinforcing their worst writing habits? And so forth.

One of the Literature Workshop’s virtues is Blau’s decision to honor the experience and practitioners’ wisdom of the English teachers he addresses, whether these are teaching in secondary schools, community colleges, SLACs or research universities.  This stance derives from the book’s origins in the National Writing Project at UC Berkeley, and its founders’ decision that its professional development would present “pedagogical ideas to colleagues largely in the form of demonstration lessons that model actual classroom practices, and then in reflecting on those demonstrations and their origins as a way of drawing a rationale or theory for practice from the demonstrated practices themselves” (15).

This characteristic integration of practice, reflection, and theorization into a recursive process honors the NWP motto of “teachers teaching teachers,” and it avoids the typical professional development scenario where teachers have to sit through lectures by outside experts or consultants who often have no concrete experience with the types of teaching being done by their audiences (17).  The danger of this kind of top-down, tone-deaf “professional development” is that it often comes from administrators or staff who have never, or no longer, face the same daily teaching challenges that faculty do, and who therefore lack credibility.  For this reason, NWP presenters like Blau only present on strategies that they themselves have practiced and refined in their own classes for an extended period of time. Convincing experienced teachers to do the uncomfortable work of reexamining what works, and acknowledging what doesn’t work, in their own teaching requires the “teacher who teaches teachers” to have some insight into the potential difficulties of such a process. For this reason, the credibility of this kind of teacher is essential for the process to have any lasting or deep effect on its audience.

There are two interesting consequences of this experiential approach to pedagogy: the first is that the kind of generic “universal teaching manual” for a particular kind of class or discipline (along the lines of McKeachie or Lang, as good as they are) seems impossible, since even the most dedicated innovator will be working in a relatively restricted curricular area.  In other words, the experience grounding a particular faculty-member’s teaching strategies will remain bound up with the type of school, student, discipline, course, and curricula that gave it a context to begin with.  This is by no means a bad thing, but it does mean that any account of teaching is deeply contextual, and requires that context to be understood in order for its lessons to be learned and implemented in a useful way.

The other consequence is that Blau’s version of English literary studies remains, in his words, “anti-theoretical,” in the sense that he is most interested in encouraging students in his introductory literature courses (and in his graduate courses for English Education students) to read and discuss literary works–at least initially– in their own terms.  He encourages his students to pursue their own lines of inquiry without having to be lectured in historical or theoretical contexts beforehand, and allows theory to “break out” (in Graff’s terms), when “agreement about such terms as text, reading, history, interpretation, tradition, and literature, can no longer be taken for granted, so that their meanings have to be formulated and debated” (5).

The result is not that theory is denied or disavowed, but that it does remain tacit, external, and undeveloped until students themselves can be brought to understand the implications of their own interpretations and interpretive debates.  This approach to theory, which makes eminent sense given Blau’s own teaching audience and courses, means that Blau’s work has less to say about other kinds of work in the English major, particularly historically-, theoretically-, or research-based-work in more advanced courses.  However, I still think there is much to learn from Blau in terms of eliciting responses from every level of student, and especially in leading discussions and designing assignments.  I’ll discuss these in another post.

DM

principles of literary history

  • For literary history to fulfill some role beyond summary, it needs to reshape, reorganize, reintegrate existing literary histories, so that what is old and what is new make better sense together side by side.
  • The pursuit of greater complexity or comprehensiveness is never sufficient reason to justify a new literary history. Instead, these become the means by which we reach a new perspective on existing writing and its histories, while introducing new materials into our thinking about literature.
  • The value of a new approach, as opposed to a new thematics, is that it should entail a truly new and different way of thinking about the material.  So how do your methods and procedures lead to different ways of thinking? Why is this change important?
  • It’s not just about the value of a particular piece of writing, but about communicating that value to someone who has never considered the writing that way before. What do they need to know, to follow you in your valuation?

DM

stalemate UPDATED

This morning I saw a post from Paul Corrigan about the assessment movement’s real impact, which amounted to “widely observed rituals of compliance” but little genuine change. The real focus of both the post and the Ted Marchese essay it took its title from was the continuing stalemate between assessment and accountability in higher ed.  This is caused in part by everyone talking past one another. Assessment experts tend to regard their own activity as a scholarly enterprise that unaccountably gets abused by the administrators who implement it. Faculty hear most assessment talk as either meaningless College of Ed jargon or administrators’ pernicious attempts to micromanage the work conducted in  classrooms.  Administrators regard it chiefly as something done to satisfy trustees or politicians, and try to think of it as little as possible otherwise. So yes, no one understands anyone else here, but that’s not why the stalemate has lasted almost as long as the assessment movement itself.

What Corrigan doesn’t seem to recognize is that these three groups do not have equal voice in this matter, because it is the administrators, as the folks who hire the assessment experts as staffers or consultants, and who “manage” the faculty, who have decided time and again to define and pursue assessment largely as accountability, standardization, and outward compliance.  There is a political economy to the way that higher education evaluates itself, and I believe that both assessment experts and disciplinary faculty need to understand how assessment and accountability both work within the emerging regimes of neoliberal management of public higher education.

Christopher Newfield, in the important piece I just linked to, spells out the strange imperviousness of administrators to the knowledge extracted by the accountability schemes they use to manage faculty and student interactions. Their imperviousness derives from their recent self-definition as managers rather than faculty members:

In contrast to professional authority, which is grounded in expertise and expert communities, managerial authority flows from its ties to owners and is formally independent of expertise.  Management obviously needs to be competent, but competence seems no longer to require either substantive expertise with the firm’s products or meaningful contact with employees.  The absence of contact with and substantive knowledge of core activities, in managerial culture, function as an operational strength.  In universities, faculty administrators lose effectiveness when they are seen as too close to the faculty to make tough decisions.

In the upside-down world of managerial culture and Christensen’s fantasies of “disruption,” paying too close attention to the information collected by others, or seeming too responsive to what it tells you about students or faculty, all these are signs of weakness, not strength.  And how else can we read the last 10 years of developments in public higher education, except as a demonstration of these principles in action?

So how might we redirect the discussion back toward improvements in learning, for both students and faculty?  One possibility suggested by Newfield is to tie improvement back to the notion of shared governance, and regard good governance and faculty communities of expertise as a necessary but not sufficient condition for improved teaching and learning.  And while we’re discussing research, I would love to see someone analyze the impact that governance has on teaching and learning.

DM

(NOTE: Those interested in discipline-specific approaches to assessment in literature departments should simply go to Laura Rosenthal and Donna Heiland’s Teagle Foundation collection to see a full range of responses to this problem)

(UPDATE, NOTE: Dr. Randi Gray Kristensen directed me to this article, which laid out a similar argument in 1999: Cris Shore and Susan Wright, “Audit Culture and Anthropology: Neo-Liberalism in British Higher Education,” The Journal of the Royal Anthropological Institute, Vol. 5, No. 4 (Dec., 1999), pp.557-575;  http://www.jstor.org/stable/2661148 )

(2nd UPDATE, NOTE: Also found this, an illuminating comparative, ethnographic discussion of “audit culture” and “neoliberalism” in various national contexts: ANDREW B. KIPNIS, “Audit cultures: Neoliberal governmentality, socialist legacy, or technologies of governing?” American Ethnologist, Volume 35, Issue 2, pages May 2008: 275–289; http://onlinelibrary.wiley.com./doi/10.1111/j.1548-1425.2008.00034.x/full)