Notes

Outline

A Few Good Measures The Impossible Dream? Presentation by Victor Borden AAHE Assessment Conference 2000 Charlotte, North Carolina – June 16, 2000

Purpose To determine together whether it is really possible to have a few good measures If it isn’t, how, when, and where can performance measures be useful? To explore some organizing concepts that may be useful in developing performance measures To leave you feeling good about yourself, because no one else has figured it out either

Method Entertainment: Opening Theme Engagement: Role Playing—What for Whom? Enlightenment: Ten years in 15 minutes Resolution

Focus and Philosophy Focus is on development and use of performance measures, not reaction to indicators mandated from external sources The best defense is a good offense

Entertainment?  - Opening Theme I look at my email, a message from way on high. The Chancellor wants measures that all our constituents will buy. To please the Board of Trustees.  State legislators of every sort. That will also improve our ranking in U.S. News & World Report. I whip out my palmtop remote access to the PeopleSoft Data Mart. Download value-added learning outcomes and that was just a start. Faculty Workload: Courses taught, grants obtained, and awards. The impact of our civic engagement on communities that we support But it was just my ‘magination.  Running away with me. It was just my ‘magination.  Running away with me.

Entertainment?  - Opening Theme Soon there’ll be measures on which we can all agree. All our constituents and stakeholders will clearly see. That faculty do more than just work for themselves. And institutional researchers write reports that don’t just sit on shelves But it was just my ‘magination.  Running away with me. It was just my ‘magination.  Running away with me.

Definition of PIs Telling someone something about What it is you do How well you do it How you might do it better Quantitative Measure

Telling Something to Somebody Unit level: department, college/university, state system Target audience: faculty at your institution, prospective students, state legislator Role: target audience – What kinds of indicators would interest you? Administrator – What do you want to indicate to the audience?

Enlightenment: Ten Years in 15 minutes Early lessons on measurement theory 1994 NDIR Volume Measuring Institutional Performance Outcomes (APQC-MIPO) Developing campus PIs to link planning, budgeting, evaluation and improvement UUPP: A multi-Institutional search for shared meaning Emerging concepts and organizers

Measurement – Theory Inductive – Deductive Cycle

Measurement - Theory Remember validity and reliability? Unless very careful attention is paid to one’s theoretical assumptions and conceptual apparatus, no array of statistical techniques will suffice – Blalock, 1982, p.9 i.e., garbage in, garbage out e.g., graduation rate, funding per FTE, etc.

1994 NDIR Volume Using Performance Indicators to Guide Strategic Decision Making (Borden and Banta, Eds.)

Lessons Borden and Bottrill: Where you stand on PIs depends on where you sit Ewell and Jones: Think before you count Joengblood and Westerheijden (Europe): PIs out, Quality Assurance in Dorris and Teeter (TQM): PIs are fine, if P stands for Process Dolence and Norris: KPIs are the fuel of a strategic decision engine DeHayes and Lovrinic (ABC): Show me the money…and what you use it for doing.

Lessons (continued) Banta and Borden Criteria for Effective PIs Start with purpose Align throughout organization Align across input, process, output Coordinate a variety of methods Use PIs in decision making

Measuring Institutional Performance Outcomes An American Productivity and Quality Center (APQC) benchmarking study

APQC MIPO Findings The best institutional performance measures communicate the institution’s core values Good institutional performance measures are carefully chosen,  reviewed frequently, and point to action to be taken on results External requirements and pressures can be extremely useful as starting points for developing institutional performance measurement systems Performance measures are best used as “problem detectors” to identify areas for management attention and further exploration Clear linkages between performance measures and resource allocation are critical, but the best linkages are indirect (and non-punitive)

MIPO Cont. Performance measures must be publicly available, visible, and consistent across the organization Performance measures are best considered in the context of a wider transformation of organizational culture Organizational cultures supportive of performance measures take time to develop, require considerable “socialization” of the organization’s members, and are enhanced by stable leadership Performance measures change the role of managers and the ways in which they manage

MIPO – Boiling it Down You cannot ‘lead’ with performance measures Performance measures emerge from a broader culture of evidence, that is, they are part of something bigger There probably is not a single set of a few good measures If a single set does emerge, it belies extensive information-based evaluation processes

Developing PIs at a Large, Complex University I was so much older then, I’m younger than that now – Bob Dylan You can’t spell IUPUI without PIs Early efforts – The laundry list approach Middle efforts – Performance Report Latest efforts – Core Mission and Core Indicators as the tip of the iceberg

The Laundry List Approach PIs by committee, prior to actually collecting data Designed around five planning themes, each having Student Learning Responsibilities of Excellence Centrality and Community Connections Collaboration Accountability and Best Practice What constitutes a credible measure The List

Performance Report Measures as part of a narrative on progress on institutional goals Textually very heavy Indicators unbalanced But it’s getting much better

Core Mission – Core Indicators From five to three themes Effective Student Learning Excellent Research and Scholarship Exemplary Civic Engagement

Effective Student Learning Enrollment Management Goals Quality and diversity of incoming students by group Retention, DFW rates, student satisfaction Transfer articulation and success Student Learning Measures Assessment results Licensure rates for professional programs Students self-perceptions of learning gains Employer ratings of skills and competencies

Excellent Research and Scholarship Grants and funding Awards and recognitions National Centers of Excellence ISI database Impact?

Exemplary Civic Engagement Contributions through education and research Elements of curriculum that use the city Retention of graduates in the area Research on ‘urban’ topics and with urban partners Economic impact

Exemplary Civic Engagement Impact of professional practice,  collaborations, and partnerships Bringle model of assessing scholarship of service Impact – effectiveness, efficiency, and significance Intellectual work – command of expertise, innovation, interdiscipliny Sustaining contribution – developmental complexity, leadership as catalyst Communication – multiple and diverse professional modes

UUPP – Six Institutions Search for Common Meaning Portfolios as indicators Indicators in the context of portfolios Example (note development version was shown at AAHE, this takes you to current production version) Emergent themes and indicator development Urban University Statistical Portrait Project will assess current measures available to address core themes

The UUPP Working Themes Access and Support Diversity and Pluralism Career and Professional Development The Urban Context for Student Learning The Urban University as an Intellectual Resource Civic Engagement

Variations on a Theme Diversity …ethnic/racial …age/roles …levels of preparation …programs and faculty Impact on learning and ability to apply learning Impact on civic engagement

Emerging Concepts Dashboard indicators Balanced scorecard

Dashboard Indicators Who’s driving? Front Seat v. Back Seat Where are we going? Command and control framework Improvement or accountability? Viability indicators as warning lights

Balanced Scorecard Toward a Balanced Scorecard for Higher Education: Rethinking the College and University Excellence Indicators Framework Brent D. Ruben, Sponsored by The Hunter Group, 2000 http://www.hunter-group.com/WhitePaper.asp Kaplan and Norton BSC Financial, internal process, learning and growth, and customer

Balanced Scorecard

Enterprise Performance Management

Indicator Blues

Ruben: Inadequacy of Accounting-Based Measures Too historical Lack predictive power Reward the wrong behavior Focus on inputs and not outputs Do not capture key business changes until it is too late Relate to functions rather than cross-functional processes Give inadequate attention to difficult-to-quantify resources, such as intellectual capital

Ruben’s Dashboard Indicators Five Indicator Clusters Teaching/Learning Programs/Courses Student Outcomes Scholarship/Research Productivity Impact Service/Outreach Extent to which university, unit or program addresses the needs of key external stakeholders according to their criteria Workplace Satisfaction Financial

Resolution What for whom PIs as the tip of the iceberg

A Few Good Measures are Possible if… You have a very focused mission and targeted clientele For the rest of us, They are designed using conceptually and methodologically sound procedures NDIR/MIPO/Balanced Scorecard criteria: mission related; reflect core values; alignment through organization and across product cycle; etc. They emerge from a broader culture of evidence They are developed for targeted audiences

What for Whom?

Questions