Category Archives: Assessment/Accountability

What Matters in Humanities Education

Since we like to talk about teaching here too, readers might be interested in my report on the Teagle Foundation‘s convening on “What Works and What Matters in Student Learning.” The conference specifically addressed the humanities. This link will take you to the web page for the event; scroll down a bit for reports by me and by Ashley Finley of AAC&U. I welcome comments and feedback on what you think matters.

fighting the last war: UPDATED

[image from http://en.wikipedia.org/wiki/Maginot_Line%5D

The new AAUP Journal of Academic Freedom came out this week, and I was happy to see that David Porter, a Psych professor and former administrator, had responded to the journal’s earlier essays by Powell and Champagne about assessment.

Porter does a pretty good job of countering the caricatured version of assessment found in Powell and Champagne, when he uses his own experience as a faculty member, administrator, and accreditation liaison to argue that

1) assessment is an integral part of learning (and hence education), 2) assessment is a necessary function of effective and adaptive organizations, and 3) involvement in assessment activities is particularly important for the AAUP and its members. Assessment is about creating a culture of evidence that is much more than merely collecting piles of data and accumulating a multitude of meaningless measures . . . (3) [my emphasis]

Porter moves the discussion from Powell and Champagne’s “culture wars,” where the humanities are always besieged, to a “culture of evidence,” in which social scientists bravely, carefully, patiently do battle with a dominant culture’s myopia to correct educational institutions.  This is a smart rhetorical move, since it raises the question of who is truly the insider and who the outsider here, and who is really representing the interests of Truth in these debates.  But is this really a debate worth having?

In some sense, we haven’t really gotten past the problem of mutual disciplinary caricatures and endless reaffirmation of tribal identities, when what we really want to know is why assessment so often fails to attain its stated institutional and educational goals.  Powell and Champagne ignore assessment as it is professionally practiced, in order to affirm the superiority of the humanities over the social sciences (e.g., Powell 12; Champagne 15). Porter, perhaps encouraged by his interlocutors’ crude self-descriptions, does some fingerpainting to dismiss Freud and Marx and their followers as insufficiently “scientific” (where’s the falsifiability?) (or, more tellingly, excessively interested in “conflict”) (18-19).

In neither Porter’s nor the Powell/Champagne approach do I find an acknowledgement that the market forces overtaking the contemporary university are affecting all its component disciplines, or that the reflection, and action consequent on reflection, necessary to contest this problem will probably need to be similarly comprehensive. Why would anyone concerned about a problem of this scale rule out of court potential allies and the kinds of evidence they could bring to the larger debate?

Finally, and this is a point I think I owe to my debates with my rhet/comp colleague JZ in my own department, I think that that much of the anti-assessment discourse in the humanities is indebted to the longstanding left critique of bureaucratic rationality from the ’60s and ’70s.  (And, indeed, much of this critique still holds true, in particular contexts; cf., for example, this).

But at this point in US history, after several decades of political reaction to the Great Society, do we really believe that the reason for non-responsive government is that favorite of conservatives and neo-libs: i.e., “bloated bureaucracy”?  Or is it the capture of once-public institutions by unaccountable financial and corporate interests? This is why I think that much of the anti-assessment discourse in Powell/Champagne’s work is a Maginot Line designed to defend us against a political opponent who no longer exists.

If we look at the intertwined history of assessment and accountability from  the vantage-point of post-Occupy American politics, what we would say is that the accountability movement, in its obsession with finding “efficiencies” in public higher ed spending, has used the long-term disinvestment in public institutions by state and federal governments to weaken internal governance and to force universities to partner with corporate interests at every level. In the meantime, neither movement has developed any cogent response to the single biggest threat that I see to the future of higher education in this country, which is the mounting student debt load necessary to attain a college degree.

So public costs are a matter of obsessive public debate, and these debates simply reinforce the drive toward disinvestment. In the meantime, the leaders of our so-called educational establishment have nothing to say about the growing alliance of finance and higher education, or how this privatization is slowly undermining the public credibility of educational institutions.

Apparently, as long as students and families are paying ever-growing amounts to private entities to gain an education, no state legislators or administrators can be bothered to intervene. So how much money would you contribute to a school run by Mark Yudof? Or Linda Katehi?  (And, indeed, no accountability advocate I’ve read seems to worry about how efficiency and privatization might impact US higher ed’s unique ties to philanthropic giving, which is unparalleled in other national systems)

In the context of the incredible debt-load that students now carry, we can now safely assume that the accountability movement has done nothing to reduce costs to students (as opposed to taxpayers), since reducing public investments in higher ed has only driven students and their families into the maw of  the (under-regulated) for-profit schools and the financial industry.

As far as I’m concerned, no savings, no give-backs wrung out of public institutions and their labor force can keep pace with the disinvestment accumulated over 30 years, and now capped off by falling state revenues during our Great Recession.  And as Krugman has pointed out in the context of the Austerity Debates, the rhetoric of controlling costs is not about “debts or deficits at all; it’s about using deficit panic as an excuse to dismantle social programs.”

So what is the way forward with accountability and assessment, under the conditions I’ve just outlined?  I’ve published some thoughts about this before, so I won’t rehash, but I think that any defense of the contemporary research university will have to take in the perspectives of all the disciplines housed inside it, as well as the full range of students we teach, and will have to develop a professional, reflective, publicly accessible view of its teaching activities to be communicated to the multiple publics we address.  Porter’s emphasis on “learning organizations” (11) sounds like a variation on my own proposal of Schon’s “organizational learning” in the university context, but I think it will have to take place in the criss-crossing, sometimes irritating discursive loop-de-loops and cul-de-sacs we are all familiar with in academic life.  In other words, no reason to over-idealize academic debate in the context of our disciplinary “tribes.”

I also think that if Porter wishes for assessment to gain the trust and respect necessary for it to be successfully integrated into our “learning organizations,” the assessment experts will need to take some responsibility for the conditions under which it is imposed.  When he writes,

Rather than building community, the hierarchical imposition of externally generated, but ill-conceived, assessment programs and inadequate protocols fractured academic communities. These initiatives also distracted and frustrated educators throughout the organization. Most of these problems, however, appear to be a consequence of the misuse of assessment principles and processes rather than deficiencies in assessment itself. (21-22)

So what ethical and professional responsibility do assessment experts like Porter take when their processes or results are misused by upper administrators or external stakeholders?  In what ways do they defend the scientific rationality of their enterprise or the deliberative process around the dissemination of its results?  How much do assessment experts attend to the organizational learning of the units whose behavior they analyze?  And how reflectively do they analyze their own role in the inevitable distributions of power and hierarchy throughout educational institutions?

In effect, to call his (and his tribe’s) long-term goal the installation of a “culture of evidence” gives the game away, because it means that the goal is not so much about instilling a respect for universal scientific reason across the university community, but trying to see whether his specific culture’s goals and values might be shared by the others in the university.  And I don’t see any reason why they shouldn’t, so long as we felt that they helped to strengthen the university as it pursues its mission.

Trying to defend not just one’s discipline but the contemporary university, as mixed as it is, means making some attempt to understand the other disciplines’ practices and what they are attempting to do; it also means developing a professional and reflective view of one’s own instructional activities and curriculum, so that students can begin to understand your discipline’s knowledge structures; finally, it means having some account of how your discipline and your university are affecting the lives and livelihoods of the people who help to support you.  How unreasonable does that sound?

DM

UPDATES:

  • If you want a really enraging example of how privatizing a public good completely distorted it, check out this piece by Andrew Leonard about Sallie Mae’s role in the student loan debacle.
  • According to this piece in College Guide, Iowa is the latest state to devise new budgetary gimmicks to reduce its support of in-state college students.
  • Sherman Dorn offers a much better and more patient demolition of David Brooks (sorry, no link for you, DB) on accountability than I could ever do, and gets into some of the real complexities it poses, especially for accreditation.

If others have additional links or sources they’d like to bring into the discussion, put them into the comments and I’ll put them up here. Thanks, DM

the politics of accountability: between the right and a hard place

At some point, I’d like to take on some more 18th-centuryish topics, but I noticed this week that the issues Laura and I have been discussing about accountability have been turning up in the presidential race.  [Warning: very little eighteenth-century content in what follows]

I would say that these discussions are occurring because accountability is the language both parties adopt when they want to politicize higher ed for their own partisan ends.  However, their uses of this language are not symmetrical.  The Right uses accountability to advance its culture war strategy against its ideological enemies and against reality itself, while what’s his name, our current post-post-Partisan Democratic President, uses it as part of his usual triangulation strategy against the constituencies that helped vote him into office (cf. Rahm Emanuel on the GM rescue: “F@ck the UAW”).

It’s not a surprise when we see Rick Santorum saying

The indoctrination that occurs in American universities is one of the keys to the left holding and maintaining power in America. And it is indoctrination. If it was the other way around, the ACLU would be out there making sure that there wasn’t one penny of government dollars going to colleges and universities, right?”

And, of course, for Santorum, Obama’s calling for all children to receive college education is a form of “snobbery,” and a veiled attempt to destroy the religious beliefs of students.  Anti-intellectualism is just one of the ways that the right wing traditionally does its version of class politics, even as we see teachers (both K-12 and higher ed, along with other public workers) getting downsized, pauperized and demonized across the country. But to some extent, this kind of attack seems very familiar to us, even when we see massive amounts of Koch money bankrolling it.

This brings me to the soggy birthday cake of ravished hopes that Obama brought with him to the University of Michigan this week, where relatively unimpoverished, undespairing students were delighted to hear that help was on the way, in the form of more work-study hours (yay, xeroxing!), more student loans, and lots and lots of accountability measures to punish universities that teach poor and working-class students.  And I’m glad that there was no mention of for-profit institutions or declining state contributions to universities, because those would just make people angrier and more partisan than they already are.

As someone who shares Laura’s hope that discussions of higher education policy become more reflective, and more effectively engaged with reality, on both sides, I’m sympathetic to her suggestion that academics think more analytically about accountability.  We do need to recognize that accountability is not an isolated aspect of our work, but something that permeates our multiple roles as scholars, teachers, and (sometime) public intellectuals.  But I do believe that part of the anger we display every time this issue comes up comes from our sense of accountability’s duplicity and hidden agendas in the wider political context.

It’s not that different than this example of Matt Damon getting irritated at being asked leading questions by an interviewer who just loves the idea of job insecurity for other people:

So how to respond to the whole context of accountability?  And what kinds of accountability can we call upon to alert the public to the dismantling of public higher education?

DM

a parable about change in higher education: michael quinn patton shares a story with us

I recall one human service program in particular where we were asked to evaluate the staff development component of the program.  In accordance with Peter’s Principle, the person in charge of staff development had risen to her own level of incompetence: she was tenured, she had territoriality on that component of the program, she could not be fired and there was no place to which to promote her; she seemed likely to be impervious to change.  No one wanted to know what staff, clients, or administrators thought about her–that was data they did not want and could not use.  We focused instead on concrete, changeable program activities (e.g., frequency and length of training sessions, content of sessions, participant input, style of training, use of outside resources, and so on).  (Evaluation-Focused Evaluation, 1978, p. 85)

[NB: the link is to the 2008, 4th edition of the book rather, than the one I quote from here. Sorry for any inconvenience this causes]

This story is prefaced by an important observation about the distinction in purpose between personnel and program evaluations: “Personnel evaluations involve gathering information about the performance of individuals. Program evaluations focus on structural and treatment characteristics of programs.  At times there is a narrow line between the two because personnel performance can, of course, affect program effectiveness.”

Most of the accountability schemes I’ve seen focus almost entirely on the personnel dimension (if only we could fire lazy professors! or eliminate tenure! etc. etc.), without acknowledging just how few options chairs and administrators have it comes to dealing with this kind of behavior.

Yet I would argue that what outside constituencies should really worry about are the program- and institutional level evaluations. Do these groups ever ask for, or see such information? After all, long after the staffer named in Patton’s anecdote retired, I can imagine a series of hiring and managerial decisions that would perpetuate her incompetence even after she was gone.  So personnel decisions have their own kind of consequence and timeline, but what do we do about a chain of ineptitude that seems to stretch all the way into the future?

So how does change take place in higher ed organizations?

DM

michael quinn patton unknowingly addresses the assessment debates in higher education, and tells us why accountability data is (almost) never used:

I’m having enormous fun with this classic argument about the hows and whys of program evaluation, which has lots of implications for higher education’s experience of “accountability”:

[Patton, Utilization-focused Evaluation, p. 88]

I’m impressed by the fact that Patton in 1978 is chiding fellow-evaluators for ignoring political and personal factors that help determine the shape, direction, and use of their studies. He argues that because evaluators would rather imagine themselves as “scientific researchers” rather than participants in a political process, they engage in a process that wastes the time of all involved, and ensures that no one uses the information gathered.  Even if the present generation of evaluators has escaped this kind of scientism,  however, it seems that many pundits, administrators, and especially politicians persist in this naive view of the role of “data” in “decision-making.”

DM