The Ideological Work of Accountability

Dave has been posting a lot about accountability, a bigger issue in Texas higher education than just about anywhere else, although it now shapes working conditions for all of us.  When I was teaching a Winter Term mini-course on “Assessment for Learning,” accountability came up a lot.  Even the quickest search on outcomes assessment in The Chronicle or Inside Higher Education will take you to debates over accountability more than any other topic.  I don’t generally write or speak about accountability, and most of the time I try not to think about it.  During the Bush years, one of my colleagues decided to pretend that Martin Sheen was the president.  That’s how I feel about accountability.

But I also recognize that you can’t get anywhere in conversations about student learning without facing this demon.  So I have been thinking about it more than usual lately, and this is what I have concluded, at least for now.

If you are an academic, it is not possible, without the gravest hypocrisy, to be opposed to accountability.  The academic project is built on accountability and has accountability at its core.

If you are a scientist, you are accountable for accurately reporting your results.  You can’t just say you performed experiments that you did not perform; you can’t just make up data.  When you do, it’s a scandal—or at least it should be, if you are caught.  Andrew Wakefield’s falsification of data that allowed him to claim a link between immunization and autism has become one such infamous case.  While the vetting process for research sometimes fails, it nevertheless serves as a (albeit imperfect) system of accountability.

If I am writing an essay on Clarissa for publication in a refereed journal, I will be held accountable for showing my familiarity with all other arguments on Clarissa that resemble mine.  Of course, if I forget to cite Terry Castle, no child will, as a result, contract a disease.  Nevertheless, most of us accept this system of accountability to limit the proliferation of essays on Clarissa that say more or less the same thing.  This accountability system has its flaws, but so far a good alternative to peer review (whether blind, open, or crowdsourced) has yet to emerge.  Digital work has brought more attention lately to this particular accountability problem, but the impulse, as far as I can tell, has been to try to figure out how to implement some kind of peer review process for digital work as well. (See, for example, 18^th Connect.)

The accountability in research, however, is only the tip of the iceberg.  We are accountable for office hours, turning in grades on time, generating credit hours, accommodating students with disabilities, affirmative action, withholding any curiosity about the personal lives of job candidates,  showing up for class, serving on committees, ordering books, grading without bias, submitting early warning grades for athletes, showing up for department meetings, holding classes that being at a given time that end at a given time, returning library books, using institutional equipment for institutional purposes.  We areevaluated every semester by the students we teach, every year by our departmental colleagues, and more or less constantly by presses, journals, and reviewers in the publication process. 

Institutions of higher education themselves are accountable for graduation rates, issuing credentials, vetting applicants, providing enough “seats” (as they say), counting the credit hours, ensuring a particular distribution of the credit hours, calculating GPAs, constructing a calendar, and reporting to their accrediting agencies.

But as Robert Barr and John Tagg pointed out more than twenty years ago, we have generally not been held accountable for student learning.  If we were, they propose, we could shed many other forms of accountability. And yet, this possibility (accountability for student learning) often comes across as outrageous, while many of those other forms do not.

Accountability, then, has a kind of ideological force.  The institutional context in which we operate renders most of our accountability invisible.  We don’t think about credit hours as a form of accountability; most of us don’t think much about them at all.  But as Barr and Tagg point out, the accumulation of credit hours tells you nothing about what student have learned. The language of “seats,” perhaps, reveals most clearly which end of the student occupies institutional attention.   The shock over accountability for student learning only suggests how far outside of the dominant ideology of accountability this function has remained.

Nevertheless, Barr and Tagg’s argument carries even more weight today.  How important are office hours, for example, when students can email with a question or to make an appointment?  I am far from the first to notice that new technologies offer the potential to change the way learning happens.  Unfortunately, though, too many of these debates focus on the wrong issues, such as whether we are “for” technology or “against” it, when we should be thinking instead is how best to use what we have to support learning.  Cathy Davidson has recently received a lot of attention, both positive and negative, for arguing that research on the brain suggests the advantages of integrating technology into learning.  Many have entered into this debate, which I won’t get into here.  I will only note, though, that a focus on cognitive science alone interestingly avoids the accountability issue by settling in advance what improves learning rather than focusing on empirical strategies that try to figure out whether or not what you’re doing is actually working.  Thus we can argue back and forth about whether student work improves more with blog posts or papers (I use both, so, as Woody Allen remarked about bisexuality, I double my chances).  But we could, alternatively, carry on this debate in the context of research (large-scale assessments), or of home-grown micro-assessments that aim to figure out which strategy works best in particular cases and for particular instructors.

So in conclusion, accountability is not the problem.  The problem is that we haven’t given enough though to which forms of accountability we would embrace (not fabricating results, accommodating students with disabilities), which are empty exercises in accountability for its own sake, which would inadvertently undermine research and learning, which are seemingly intended to undermine research and learning (hello Texas), which have been rendered vestigial by new technologies, and which should be getting more of our attention.

a parable about change in higher education: michael quinn patton shares a story with us

I recall one human service program in particular where we were asked to evaluate the staff development component of the program.  In accordance with Peter’s Principle, the person in charge of staff development had risen to her own level of incompetence: she was tenured, she had territoriality on that component of the program, she could not be fired and there was no place to which to promote her; she seemed likely to be impervious to change.  No one wanted to know what staff, clients, or administrators thought about her–that was data they did not want and could not use.  We focused instead on concrete, changeable program activities (e.g., frequency and length of training sessions, content of sessions, participant input, style of training, use of outside resources, and so on).  (Evaluation-Focused Evaluation, 1978, p. 85)

[NB: the link is to the 2008, 4th edition of the book rather, than the one I quote from here. Sorry for any inconvenience this causes]

This story is prefaced by an important observation about the distinction in purpose between personnel and program evaluations: “Personnel evaluations involve gathering information about the performance of individuals. Program evaluations focus on structural and treatment characteristics of programs.  At times there is a narrow line between the two because personnel performance can, of course, affect program effectiveness.”

Most of the accountability schemes I’ve seen focus almost entirely on the personnel dimension (if only we could fire lazy professors! or eliminate tenure! etc. etc.), without acknowledging just how few options chairs and administrators have it comes to dealing with this kind of behavior.

Yet I would argue that what outside constituencies should really worry about are the program- and institutional level evaluations. Do these groups ever ask for, or see such information? After all, long after the staffer named in Patton’s anecdote retired, I can imagine a series of hiring and managerial decisions that would perpetuate her incompetence even after she was gone.  So personnel decisions have their own kind of consequence and timeline, but what do we do about a chain of ineptitude that seems to stretch all the way into the future?

So how does change take place in higher ed organizations?

DM

Gaming Table at the Folger

Here is story in today’s Washington Post on the Folger production of Centlivre’s play (The Basset Table), in which I am quoted praising her feminism.  There are also links to the production in progress.

michael quinn patton unknowingly addresses the assessment debates in higher education, and tells us why accountability data is (almost) never used:

I’m having enormous fun with this classic argument about the hows and whys of program evaluation, which has lots of implications for higher education’s experience of “accountability”:

[Patton, Utilization-focused Evaluation, p. 88]

I’m impressed by the fact that Patton in 1978 is chiding fellow-evaluators for ignoring political and personal factors that help determine the shape, direction, and use of their studies. He argues that because evaluators would rather imagine themselves as “scientific researchers” rather than participants in a political process, they engage in a process that wastes the time of all involved, and ensures that no one uses the information gathered.  Even if the present generation of evaluators has escaped this kind of scientism,  however, it seems that many pundits, administrators, and especially politicians persist in this naive view of the role of “data” in “decision-making.”

DM

Centlivre at the Folger

Eighteenth-century theater fans can also follow the production blog for Susanna Centlivre’s The Gaming Table (originally, The Basset Table).  If you scroll down, you can find instructions on how to play basset.

The London Merchant

The Storm Theatre and  The Blackfriars Repertory Theatre in New York are now producing George Lillo’s The London Merchant!  The play received a favorable review in The New York Times.  I have never seen this play produced or even heard of a modern production, but students consistently like it. The production seems like it would be well worth seeing.

MLA 2012: The Future of Early British Studies

A Marketplace of Ideas? The Future of Early British Literary Studies

Presiding: Robyn Malo, Purdue Univ., West Lafayette; Manushag Powell, Purdue Univ., West Lafayette

1. “Problems for the Future,” Helen Deutsch, Univ. of California, Los Angeles

2. “Curricular Requirements and the Problems of the Present,” Seth Lerer, Univ. of California, San Diego

3. “Solutions?” Emily Hodgson Anderson, Univ. of Southern California

The subject of this panel was the challenges and opportunities facing Early British Studies in the current climate. What kind of future does Early British Studies have in higher education?  How can we engage students?  The panelists considered these questions in various ways.

Helen Deutsch looked back in order to look forward, we might say.  She implicitly argued against the suspicion that Early British Studies have no relevance to what people care about now.  Her strategy was to demonstrate in fascinating detail the influence of Jonathan Swift over Edward Said.  She reminded us that Said had long planned a book on Swift; she suggested the profound connections between the kind of public intellectual that Said became and vigorous eighteenth-century models for such a position.

The next two panelists focused on student engagement with the period.  Seth Lerer discussed the challenges of teaching Early British Literature to a new generation of students.  He described a large lecture class he was teaching at San Diego, in which the majority of students spoke English as a second language and only one had brought the book.  The rest were reading the material on their iPads, laptops, and even iPhones.  Yet in spite of this set-up, the talk did not turn curmudgeonly.  These students were welcome on his lawn, and he took seriously the challenge of communicating with them.  He proposed that we include the history of technology in the way we teach Early British Literature, drawing connections between the move to the digital and the transition to the codex.  He argued that this kind of contextualized narrative would be consistent with the discipline itself, suggesting that one of the distinguishing characteristic of humanities disciplines was concern with its own history.  The sciences, he pointed out, supersede their history and thus have little interest in what came before.

Finally, Emily Anderson offered some thoughts about the problem of “relevance.”  She noted tensions in eighteenth-century courses between our impulse to historicize and the student desire to find themselves in the literature, collapsing those historical differences.  She pointed out that students often come to literature classes out of a desire to write their own story.  Her strategy, which she has found to be effective, has been to use this to her advantage and cultivate this impulse, but then also, we might say, to theorize the impulse itself.  For this she uses Tristram Shandy, though a difficult text for undergraduates, as a model, which is after all the story of someone writing himself into being. She has even started to offer students a creative option to the usual critical paper, although they also need to discuss their choices and strategies in a critical way.

Overall, a worthy and engaging panel, filled with great ideas about how to bring Early British Studies into the 21^st Century.

LR

MLA 2012: When Assessment Goes Bad

[x-posted at http://assessmentforlearning101.wordpress.com/]

On the first day of MLA 2012 I attended “Assessing Assessment(s),” chaired by Jeanne A Follansbee (Harvard), with talks by Donna Heiland (Teagle Foundation), John M. Ulrich (Mansfield University), and Eve Marie Wiederhold (George Mason). Reed Way Dasenbrock was unable to attend, which is a shame because I heard an excellent talk that he gave last year and was looking forward to his perspective on this issue. (I have also taught his essay from Falling into Theory in my “Critical Methods in Literary Study” class.)

All the papers were sharp and interesting, with Heiland considering the role of assessment in cultivating student learning, Ulrich reporting on the highs and lows of his institutional practice, and Wiederhold offering a vigorous critique.

But what really enlightened me at the panel was the Q&A, during which it became clear that there was a lot of really terrible assessment going on out there. One speaker described how an “assessment professional” had been hired at her institution to set the learning outcome goals for all the programs. Another reported that he regularly turned in a series of graphs charting student grades, much to the delight of local assessment administrators.

I had mostly assumed that everyone hated assessment because it is part of the paradigm shift described by Tagg and Barr from “Instruction” to “Learning” (a point discussed by Heiland) which pretty radically goes against the status quo and thus makes people anxious. (Maybe this goes back to Dave’s discussions of “threshold concepts.”) Further, I too hated it at first, as it seemed redundant and intrusive. Now, though, I see it as part of a potential change from counting credit hours (or as my former provost used to say, “butts in seats”) or relying on student evaluations (or, as Roksa calls them, “student satisfaction surveys”) to opening up new ways of emphasizing, appreciating, and thinking about learning itself as the goal, which in turn leads to thinking that there might be better ways to get there than counting up things up, be they credit hours or survey scores. So while assessment has the reputation of bean counting, in fact we are currently wading through heaps of beans (credit hours; evaluation scores; grades; office hours; chairs bolted to the floor; multiple choice tests) without even noticing them as they have become so natural to our environment. In a true “culture of assessment,” there would be fewer beans.

It seems, though, at some institutions assessment has not been part of a larger consideration of student learning, but instead the evil bureaucratic exercise that many feared it would become.

Lisa Gueldenzoph Snyder, “TEACHING TEAMS ABOUT TEAMWORK: PREPARATION, PRACTICE, AND PERFORMANCE REVIEW”

This article should be available to any library that subscribes to SAGE journals online.  It stresses that collaboration is something that needs to be taught and practiced in formal courses, and divides up the instructor’s duties into the stages of preparation, practice, and performance review.

Especially helpful are these recommendations for keeping in-class group activities on track:

• Focus attention on the purpose of the project: In small groups, ask students to brainstorm methods of refocusing a group discussion. Ask students to role-play reactions to statements and list the methods in order of effectiveness.
• Encourage participation and positive collaboration: Address active listening, questioning, and restating techniques to ensure that students participate and provide input during group discussions. Encourage students to engage in positive collaboration among team members and
  referee any unconstructive feedback or personality conflicts.
• Establish a timeline: Demonstrate how to create a timeline by working backward from a deadline. Discuss delegation and prioritization techniques to ensure a balanced yet productive group experience.
• Keep the project on track: Although individual team members may
  work on separate tasks, students should be coached to schedule team meetings throughout the project to discuss progress, encourage group feedback, and share ideas. They should also be coached to accept new ideas and revisions to the plan that enhance the project (rather than perceiving changes as negative aspects that prolong the team endeavor).
• Negotiate conflicts: Ask students to role-play the differences between
  affective and cognitive conflict and practice impartial methods to resolve any problems.

Well worth checking out.

DM

Why do students hate groupwork? Part 2

[X-posted on Assessment for Learning 101]

One of the most-read posts on this blog is David Mazella’s classic, “Why do students hate groupwork?”  The original post prompted a lively discussion, including comments by students themselves telling us why, in fact, they hate groupwork.

I was thinking about this discussion yesterday at my MLA panel, “Academically Adrift,” which featured Josipa Roksa, one of the authors of the book after which the panel took its name.  Offering a brief overview of her findings (with co-author Richard Arum), Josipa introduced the section on study groups by saying, “This is the one that always gets me in trouble.”  And indeed, part of the discussion that followed concerned this issue.

In short, Arum and Roksa found that students who worked in study groups showed significantly lower learning than those who studied on their own.  What does this mean for collaboration?

To me, the most interesting point that came out of the conversation was that the problem might not be group work itself, but the way it is done.  Roksa speculated that there is a tendency for the professor to assign group work without enough structure and also without providing any training for students in collaboration itself.  We tend to come up with a project, give it to a group of students, and say “go collaborate,” which turns out to be ineffective.  She briefly discussed a colleague of hers who teaches a semester-long course specifically on collaboration.  So perhaps her answer to the question posed by Dave’s original post might be that students hate it because they don’t know how to do it and as a result they don’t learn much.

What I found particularly interesting was Roksa’s emphasis on collaboration as a skill that needs to be learned.  As someone raised on theory that taught me how gender, race, and all kinds of identity formations are constructed, I had never given much thought to collaboration as “constructed” as well.  Perhaps for too many of us, it seems like something that students should just know how to do.  But apparently they don’t (and in fairness, we often don’t either).  I wonder, then, if we should be thinking about ways to get collaboration skills into the curriculum—not just in the form of collaborative assignments but as a learning outcome goal in itself.

LR