The Urban Universities Portfolio Project (UUPP) formally concluded its work in June 2001. This report focuses on major developments and activities from July 2000 through June 2001, the third and final year of the project, reports on the project's overall accomplishments and progress on key deliverables, and discusses lessons learned over the course of the project. The article on the UUPP that appeared in the May/June 2001 issue of Change magazine and the section on electronic institutional portfolios in the 2001 AAHE volume, Electronic Portfolios: Emerging Practices in Student, Faculty, and Institutional Learning, incorporate much of the collective learning of the project about the value of electronic institutional portfolios for campuses and their stakeholders, processes for developing and using such portfolios, and the role of interinstitutional collaboration in shaping the work of individual campuses and, in turn, being shaped by that work.

In the final year of the UUPP, project participants focused on developing the individual institutional portfolios to the fullest extent possible, testing the portfolios with external stakeholders, commissioning a study of the resources necessary to create and maintain an electronic institutional portfolio, and widely disseminating the work and lessons of the project through presentations and publications.  Major accomplishments and activities included the following:

Campus Work

• By the end of Project Year 3, each of the six participating campuses had completed a functional prototype for a Web-based, electronic institutional portfolio.  These prototypes represent perhaps the most important product of the project.  With no real models to guide them, the six campuses, working with internal groups and with one another, turned the portfolio concept envisioned in the original project proposal into reality.  Our first-generation prototypes have already elicited considerable attention nationally and internationally, as discussed later in this report.

• New planning, assessment, and improvement initiatives stimulated by work on the portfolios continued or were begun on all six project campuses.

• While the six campuses did not reach agreement on a common set of core indicators of effectiveness for urban public universities, or even on common characteristics of urban universities, the project’s emphasis on exploring these issues stimulated discussion and reflection on each individual UUPP campus on the ways in which its urban setting and mission have informed its character and work and influenced student learning.  Most campuses report that these discussions and reflections helped them reach a more coherent and broadly shared awareness of their institutional identities and roles.  This developing awareness is reflected in various ways in the six portfolios and is a theme of several of the UUPP-stimulated assessment and improvement initiatives on individual project campuses.  In addition, a project-generated draft on “common characteristics of urban public universities” provided a starting point for a related project, discussed below, that has made further progress on indicator development.

Meetings and Collaborative Work

• The UUPP continued its program of regular project meetings, where participants exchanged progress reports on their portfolios and related campus initiatives, gave and received feedback and suggestions, and discussed the project’s implications for a range of higher education policy issues.  Four project meetings were held in 2000-2001:  two meetings of all project participants in August 2000 at California State University, Sacramento (CSUS) and in June 2001 in Denver, in conjunction with the AAHE Assessment Conference; a meeting of all UUPP campus participants in October 2000 at the University of Massachusetts Boston (UMB), in conjunction with the Fall 2000 Urban 13 meeting; and a meeting of campus project directors in January 2001 in New Orleans, in conjunction with AACU’s annual national conference.  The impact of these meetings and of the collaboration among the six campuses and AAHE is discussed later in this report and is the subject of the attached Change magazine article.

• A draft document developed early in Project Year 3 on the defining characteristics of urban public universities provided the conceptual framework for a related collaborative effort, funded by the Urban 13 Academic Officers and the Coalition for Urban and Metropolitan Universities, to identify the measurable characteristics of urban and metropolitan universities, develop common surveys focused on key indicators, and form a data exchange consortium and a commonly accessible database.  Institutional researchers from 39 institutions are participating in this effort.  The project, called the “Urban Universities Statistical Portrait Project” in its first phase and known as the “Portrait of Universities with Metropolitan Alliances” (PUMA), has already made considerable progress; information on it can be found on the project’s Web site at www.imir.iupui.edu/puma.

Audience Testing

• In addition to the regular meetings of participants throughout the year, the project hosted a special meeting, “Urban University Portfolios:  Performance and Prospects,” at the Hayes Mansion in San Jose that brought UUPP presidents/chancellors and provosts together with representatives of key stakeholders in higher education, such as accrediting associations, state-level higher education officers, students, employers, the media, and educational policy organizations.  The purpose of the meeting was to test the electronic institutional portfolio concept and the developing portfolios themselves with the external audiences intended as users of the portfolios.  Overall responses to the portfolios and the project were extremely positive; at the same time, discussions clarified the varying interests and information needs of different user groups and highlighted several key issues, detailed later in this report, for the project and follow-up work to address. 

• Each participating campus hosted a site visit by two members of the UUPP Institutional Review Board (IRB) and one or two representatives of other project campuses.  The visits, also discussed in more detail in a later section of this report, provided additional opportunities to test the portfolios with external audiences. 

Resource Issues

• The project commissioned a study of the fiscal, human and technological resources needed to develop and maintain an electronic institutional portfolio—a “Functional Needs Assessment”—from Edutech International.  Representatives of Edutech, along with the UUPP Technology Development Associate, made site visits to the six project campuses, producing individual reports and recommendations for each campus, as well as a summary report on resource issues.  Results of this study are summarized in the “Lessons Learned” section of this report; Edutech’s project-wide report is also attached.

Dissemination

• Efforts to disseminate the work of the project through conference presentations were increased and intensified during the final project year.  Participants report a total of over 30 presentations at regional, national, and international conferences between July 1, 2000 and June 30, 2001.  These included presentations at the annual conferences of four regional accrediting associations, a plenary session at the AAHE Assessment Conference, and several pre-conference workshops aimed at helping campuses begin work on electronic institutional portfolios. 

• The project produced several publications during 2000-2001, in addition to the monthly reports on the project in the AAHE Bulletin.  These included an article in Change magazine focused on the UUPP and a related Pew project, as well as a major section of AAHE’s book, Electronic Portfolios:  Emerging Practices in Student, Faculty, and Institutional Learning.  A special issue of Metropolitan Universities on the UUPP is planned for Summer 2002.  

• The project’s Technology Development Associate continued to develop and expand the project Web site. 

Progress on Key Project Deliverables

The UUPP grant agreement with The Pew Charitable Trusts lists four key deliverables for the project.  Progress on each of these deliverables is discussed in this section.

1. The participating institutions will have a body of evidence that they can use for internal improvement.

Portfolios and “Bodies of Evidence”

The six online electronic institutional portfolios document the “body of evidence” that each campus developed over the course of the project’s three years.  (The portfolios can be accessed from the project Web site at http://www.imir.iupui.edu/portfolio/.)  The amount and characteristics of that evidence vary considerably, partly because the six campuses were at varying levels of development of assessment practices, institutional research functions, and technological capability, and partly because each campus had to come to grips in its own way with the initially abstract concept of an electronic institutional portfolio.  In the process of making this concept concrete, most campuses went through multiple iterations of the organizing frameworks for their portfolios and began new planning, assessment, or improvement initiatives to develop evidence (or additional evidence) of effectiveness for the portfolios. 

Thus, while campuses freely borrowed ideas and particular features from others’ portfolios, we found that a “one size fits all” model for an electronic institutional portfolio would not generate meaningful internal improvement in this project.  Instead, portfolio development needed to align with each institution’s distinct culture, practices, agenda for the project and intended audiences for the portfolio.  For example, several campuses were seeking a catalyst for efforts to launch or improve assessment programs, with regional accrediting associations as a primary audience.  Others, aiming mainly at internal audiences, focused on the potential of technology to gain wider campus participation in planning initiatives and to improve internal decision-making and academic program review practices.  Some were interested in finding more effective ways to articulate and communicate their missions and visions both internally and externally.  Most hoped to pursue a combination of these objectives tailored to their specific current needs.  An important finding of the UUPP is that an electronic institutional portfolio initiative can provide a conceptual framework and generate new organizational structures to support a variety of campus improvement efforts.

For example, the campus project director at Georgia State University (GSU) writes that “the Urban Universities Portfolio Project has enabled Georgia State University to rethink and restructure its processes to take advantage of the potential of the Internet.”  GSU used its portfolio Web site as a forum for campus-wide discussion of its developing strategic plan, enabling the university to “expand its participatory planning and decision-making processes” and to gain wider campus buy-in for the plan and subsequent annual action plans.  As part of an effort to reorganize its institutional research functions and to make data readily available to decision-makers, GSU also incorporated a dynamic database into its portfolio, GSU Statware, that provides internal users with access to information tailored to their specific data questions and needs.  A focus on using the portfolio to improve the accessibility of planning processes and data thus characterized the project at GSU. 

Other project campuses deliberately approached development of an institutional portfolio as an occasion to jump-start campus-level assessment of student learning outcomes.  Several of the campuses—California State University, Sacramento (CSUS), GSU, and the University of Illinois at Chicago (UIC)—adopted explicit campus-wide learning outcomes for the baccalaureate in the course of the project as a starting point for implementing more systematic approaches to assessment.   Others, such as IUPUI and PSU, strengthened and augmented already-existing assessment practices.  The campus project director at Portland State University (PSU) also credits the project with catalyzing a series of provost-led campus discussions about the institution’s vision and mission. 

Portfolios and “Cultures of Evidence”

Perhaps an even more important outcome of the UUPP is that, in addition to helping participating campuses develop “bodies of evidence” to be used for internal improvement, it also fostered “cultures of evidence” that are encouraging more systematic reliance on data and assessment results to inform planning and improvement among key groups at all six institutions.  In this sense, the project has expanded each institution’s capacity to develop and use evidence, especially evidence related to student learning, beyond the period of the formal project itself.

The campus reports presented at the project’s final meeting in June 2001 as well as the written final campus reports bear ample witness to this cultural change.  For example, IUPUI’s campus project director reports that “in order for the IUPUI Portfolio to demonstrate how [IUPUI] achieves its mission…we needed to identify what ‘bodies of evidence’ we could draw upon, and what ‘bodies of evidence’ needed to be developed.”  As a result of analyzing needs for evidence for the portfolio, IUPUI undertook a study of how, when, and where its “Principles of Undergraduate Learning” are taught, learned, and assessed in programs across the campus.  It also created representative working groups to articulate goals and develop assessment plans and methods for Civic Engagement and for Research, Scholarship, and Creative Activity, two key components of the campus’s mission.  Development of the portfolio thus provided a conceptual framework that prompted examination of where the campus lacked systematic approaches to goal-setting, assessment, and improvement in mission-critical areas.

Similarly, the campus project director at the University of Massachusetts  Boston (UMB) explains that “we are not simply taking what we have already been doing and presenting it to the public in the format of a portfolio.  This project is helping to prod us into doing what we have wanted to do, but have not had the time or resources to do.”  For example, he reports that the graphical representation of the results of the National Survey of Student Engagement in the UMB portfolio stimulated a great deal of faculty discussion of learning.  In this instance, UMB found “that a tool that was originally intended to make our case to external audiences—simple, graphic representation of students’ reports of classroom practices—has been the best way to provide information to the faculty to be used in program improvement.”  Direct results of these conversations at UMB include a new initiative to assess the impact on student learning of the university’s diversity requirement and a faculty that is “more educated about assessment.”  In this way, says the UMB campus director, “the portfolio becomes part of the feedback loop.”

Other UUPP campuses agree that creating a portfolio that represents the institution’s mission, goals, and accomplishments can stimulate new collaborative examination and discussion of how mission, goals, and the day-to-day work of the campus are related to one another and how evidence of accomplishment and fulfillment of the mission can be developed and used to improve effectiveness. The campus project director at UIC includes substantial discussion in his report about the project’s influence in this regard.  He notes that the campus, which is highly decentralized, lacked a culture of assessment or any centralized support for assessment and strategic planning at the outset of the project. The UUPP provided a context in which to bring key people from across the campus together for a “significant group of campus conversations” around student learning and assessment.

A result of these conversations at UIC was that by the third year of the project, according to the campus director, “’body of evidence’ questions came second nature to us,” while important committees and groups within the institution, including its senior leadership, were committed to the goals of assessment and to creating a new office within the central administration to provide leadership on institutional effectiveness issues.  He argues that “a nascent assessment initiative needs something like the portfolio project” as a framework and catalyst for campus-wide collaboration and action on assessment. 

All six campus reports suggest that the campus-wide collaborations required to build an electronic institutional portfolio were key to fostering cultural change around learning, improvement, and evidence.  The collective nature of the effort broke down barriers among offices, programs, and initiatives that, in many cases, had been working in isolation from one another.  For example, at several UUPP campuses, institutional research offices, which are important information resources for the portfolios, developed closer ties to faculty groups and other administrative units and became more firmly integrated into campus planning and improvement efforts.  Such integration is furthering the development of campus cultures that use data to inform planning and improvement and is also changing institutional research functions and practices.   For example, the campus project director at Portland State University (PSU), who is also Director of Institutional Research and Planning for the campus, reports that whereas IR was viewed before the project as a narrowly focused data support function, it has now become central to campus decision-making and implementation of campus improvement efforts.

2.   The institutions will have improved their capacity for public accountability.

Impact of the Project on Campuses’ Public Accountability

The development of stronger cultures of evidence at UUPP campuses and the improvement of the institutions’ capacity to gather, interpret, and act on evidence of effectiveness, discussed under Deliverable 1, is a prerequisite to improving their “capacity for public accountability.”  Several campus directors say as much in their final reports.  IUPUI’s campus project director notes that “the conversations and the need to establish the schema [for structuring the portfolio] have taken faculty and the administration much further into capacity-building for public accountability than we have been in the past.”  At GSU, development of the electronic institutional portfolio has led to plans for an electronic annual report that will be widely accessible on the Web.

In the UIC portfolio, where current and prospective students are seen as primary audiences and stakeholders for the portfolio, accountability is taking the form of “data on racial, ethnic, and gender profiles…in the current student body…[information on] time to degree, rates of student retention, opportunities for transfer students…student learning goals and…degree profiles as well as alumni surveys,” in addition to other information intended to inform prospective students about the educational experience they can expect at UIC.  For current students, the UIC portfolio will “provide content maps to a successful freshman year experience, to coherent General Education sequences, and to choice of major.”

UUPP institutions’ improved capacity to gather, interpret, and present information of effectiveness for public accountability purposes has gone hand-in-hand with their increasing willingness to include such information on a public Web site.  Early in the project, campuses voiced concerns about the consequences of making such information openly available, while project meetings included frequent and spirited debates about what we came to call “the dirty laundry issue.”  Such concerns could not be easily dismissed; the six campuses were scattered across five regional accrediting associations’ areas, were each part of a different state university system, and had to contend with varied media environments and other local conditions.

Nonetheless, over the course of the three-year project, as participants’ understanding of “institutional effectiveness” came to include the concept of ongoing assessment and improvement, and as campuses saw others in the project openly posting information about areas of relatively weak performance, the debates faded and institutions became more comfortable with making such information public.  In this sense, campuses not only increased their “capacity for public accountability,” but also actually made accountability information more publicly accessible.  One participant described his campus’s approach to the issue this way:  “when we air our ‘dirty laundry,’ we always put a washing machine [i.e., information on efforts to improve] right next to it.” 

The use of the Web as the medium for the portfolios played a key role in bringing about the results described above.  The discussion of Deliverable 4 includes detailed consideration of the influence of technology on the outcomes of the project.

Usefulness of the Portfolios for Public Accountability

To test the usefulness of the portfolios for accountability to various groups of stakeholders, the UUPP convened a small (about 25 participants) invitational meeting of chancellors/presidents and provosts from project campuses with representatives of key stakeholder groups.  Held in San Jose on October 15-17, 2000 and facilitated by leaders of the Aspen Institute Program on Education in a Changing Society, the meeting on “Urban University Portfolios:  Performance and Prospects” included participants representing the accreditation community, SHEEOs, higher education policy organizations, employers, students, and the media. 

Participants were “assigned” fairly extensive preparation for the meeting, with each asked to review one of the electronic portfolios in detail and to respond in writing to questions about its usefulness to their stakeholder group, its effectiveness in addressing student learning and the urban mission of the institution, and its clarity about the institution’s strategic priorities.  These comments, which subsequently were made available to UUPP institutions, provided a basis for wide-ranging discussions at the meeting about how the portfolios could be made more helpful, informative, and accessible to the range of stakeholders the project hoped to address.  Project leaders wanted to know, for example, what stakeholders thought about the electronic institutional portfolio concept, whether they really cared about the issues, especially student learning outcomes, that the portfolios addressed, and whether the portfolios addressed these issues in meaningful, informative ways.

Reactions to the portfolio concept were uniformly positive:  meeting participants were impressed by the self-examination and improvement processes represented and stimulated by the portfolios and lauded the UUPP campuses for focusing on student learning and for making assessment results public.  The use of authentic evidence and primary materials from students and faculty to document teaching and learning helped participants reach a fuller understanding of the realities of education at each institution.  Participants commented that the portfolios they examined gave them the sense that “we know what we’re doing at this institution.”  The “dirty laundry” issue generated extensive discussion, with participants generally concluding that universities and colleges should indeed make information about institutional and educational effectiveness publicly available, but agreeing that it would be difficult for individual institutions to step forward and do so unless many other institutions were doing the same.

Views on the usefulness of the portfolios for accountability and public policy purposes were more mixed.  Accreditation and student representatives were enthusiastic about the detailed, close-up view of an individual institution that electronic portfolios can provide.  Stakeholders who needed information about institutional effectiveness for state-level accountability or for classifying or ranking institutions found that the lack of uniformity in the structure and types of information found in the portfolios, particularly the lack of common indicators and measures of effectiveness, detracted from the usefulness of the portfolios for their purposes.  These stakeholders wanted to see more summative, quickly digested information that would allow for comparison among institutions.

In the end, the outcomes of the San Jose gathering provided valuable feedback and guidance to project leaders and campuses.  Individual campuses incorporated the comments from the pre-meeting “homework assignment” to clarify the purposes of their portfolios for viewers, identify areas where more or different content would be helpful, and simplify language and navigation schemes.  Confirmation that stakeholders were genuinely interested in college-level learning gave encouragement to campus efforts to assess and improve that learning in meaningful ways.  The project as a whole gained a clearer understanding of the accountability needs of various sets of stakeholders and of both the contributions and the limitations of the UUPP portfolios in meeting those needs.  This understanding is proving especially helpful as we work on plans for possible follow-up projects that build on the work of the UUPP to take electronic institutional portfolios to the next level of development.

3.  The project will produce a core set of indicators of effectiveness that, although they will be designed specifically for one sector of universities, can be used as a template by others.

Indicator Development Efforts

One of the original ambitions of the UUPP was to develop a description of the core mission and characteristics of urban public universities, along with a set of indicators of institutional effectiveness drawn from this description.  While project meetings included extensive discussion of this issue and yielded successive draft descriptions of potential common characteristics (see the attached document for the most recent draft), the six campuses never reached a final consensus.  In retrospect, this outcome seems predictable for several reasons:

• Most of the campuses in the project lacked internal consensus on what it means to be an urban public university.  For example, concerns that a more precise definition of the urban mission would limit the kinds of research the institution might pursue or the curricula it might offer were common among faculty and administrators at the six campuses.  Even within the individual campuses’ UUPP project teams, we found disagreement about the importance of identifying common characteristics and indicators for urban universities.
  
  
• The institutions in the project were, in fact, quite diverse, varying in size, selectivity, program mix, and levels of emphasis on research and on undergraduate and graduate education, among other factors.
  
  
• Like many other urban and metropolitan universities, the six UUPP campuses are relatively young, rapidly changing institutions.  For example, over the course of the project, several campuses became much more selective in their admissions policies, either by mandate or because of other circumstances, with student bodies moving toward more traditional, less classically “urban,” demographic profiles as a result.
  
  
• The need for each campus to create its own process for developing and structuring its portfolio in order for the effort to flourish and lead to real improvement ran counter, for some participants, to the idea that the portfolios would somehow illustrate a set of common institutional characteristics.

An insightful summary of the reasons that the project “did not produce a uniform set of indicators accepted by all the participating universities” is offered by the campus project director at UIC in his final report:

In an experimental enterprise such as this one, perhaps the experience of carrying the first group of Portfolios into relative maturity is the lesson itself.  Our accounts of procedures, false starts, regroupings, successes and evolution yields more of a narrative now than a declarative set of indicators.  It is perhaps premature to have expected six campuses embarked on such a new errand to not only carry it through on their own turf, but to have had the time and reflection necessary to produce common indicator sets.  As the campuses worked together in many meetings, it became very clear that we had disparate campus cultures that dictated different rates of development and different campus expectations and support toward what …a Portfolio might be.

Outcomes of Indicator Development Efforts

The conversations within and among the campuses about the characteristics of urban public universities were still productive in several ways.  Because the urban mission was a central theme of the project, each campus considered how its own “urban mission” informed its priorities and practices in order to represent this mission in its portfolio.  These internal discussions led to a sharper, more clearly articulated, and more widely shared understanding of mission and vision at each campus, an understanding reflected in the six portfolios and in the campuses’ final reports on the project.

For example, PSU’s campus project director explains that “PSU’s portfolio begins with vision statements from the President and Provost that address the University’s identity and mission as an urban university.  References to mission and how it is played out in our activities are threaded throughout the portfolio content,” most clearly highlighting the institution’s urban identity in the chapter on “Community and Global Connections.”  UIC’s campus director similarly notes that “a principle of inclusion in the UIC Portfolio is that each item or profile should enhance the Web site browser’s knowledge of UIC’s complex relationship to Chicago…to allow our audiences to gauge the full participatory nature of UIC’s educational commitment as an urban campus.”  GSU’s campus director reports that the portfolio introduces viewers “to a research university located in an urban setting and [provides] a multidimensional view of the engagement of the university in its communities.” 

As these and similar comments from other campus reports imply, a review of the six portfolios does indeed suggest some commonalities among the institutions’ visions of themselves as urban universities:  all stress their commitment to engagement with the urban community as a strong component of their missions; all emphasize their efforts to draw on the resources of their cities to enrich students’ educational experiences; and all make a point of highlighting the diversity of their student bodies. 

In addition, the project’s efforts to identify the characteristics of urban public universities led to the participation of five of the UUPP institutions in the first large-scale pilot of the National Survey of Student Engagement (NSSE) during the 1999-2000 academic year.  Analysis of our results suggests that our students’ educational experiences differ in important—and similar—ways from those of students at other large public research universities.  Results of the UUPP participation in the NSSE were extensively reported on in the 1999-2000 annual narrative report on the project.

Some of the individual and collective work of the project on “urban” characteristics and indicators has produced models, as well as a related follow-up project, that we hope will lead to further progress on indicator development for urban and other institutions.  IUPUI’s model, which uses green, amber, and red “traffic lights” to illustrate levels of attainment for fourteen indicators related to mission, has attracted a great deal of attention at conference presentations on the IUPUI portfolio.  The portfolio also led to a new initiative at IUPUI to develop a model for assessing effectiveness in the area of civic engagement, based on criteria that include measures of institutional capacity for effective civic engagement (such as administrative structures and practices that support faculty and student involvement in civic engagement activities), process measures (such as the quality of partnerships with community agencies), and measures of impact on both the university and the community.  The model is currently under development and is intended to provide a framework that other institutions might also use to assess the effectiveness of civic engagement.

Finally, the PUMA project, which includes institutional researchers from 39 urban and metropolitan universities, used the most recent UUPP draft document on common characteristics of urban universities as a conceptual starting point for a new effort to develop indicators of effectiveness for these institutions.  The group is drawing on several sources of data, including the National Center for Education Statistics, the Common Data Set, and several national surveys to derive a set of measurable characteristics of participating universities. 

4.  Through the example of the institutional portfolio and external review, the project will yield a more appropriate approach to quality assurance and should improve both accreditation and state review practices. In form, content, production and purpose, the portfolios and the process of reviewing them will differ significantly from current kinds of documentation used in program evaluation and institutional accreditation.

Implications for Quality Assurance

The UUPP and the six electronic institutional portfolios have significant implications for quality assurance.  Their focus on evidence and outcomes, particularly educational outcomes, on effectiveness in carrying out the specific mission of the institution, and on ongoing assessment and improvement is consonant with current national discussions of what constitutes quality in higher education.  These discussions emphasize student learning, accomplishments in relation to mission, and a commitment to continuous improvement processes, all documented by evidence, as the principal gauges of quality.  The experience of the UUPP suggests that electronic institutional portfolios can not only provide the evidence required to demonstrate quality, according to this view of it, but that they also can support the development of institutional practices that help institutions improve quality, as suggested in the discussion of Deliverable 1.   

The ultimate extent of the influence of the UUPP on accreditation and state review practices remains to be seen.  But already, all six regional accrediting associations have expressed interest in the potential of electronic portfolios to replace or supplement traditional self-studies.  The Western Association of Schools and Colleges’ Senior College Commission, whose Executive Director was on the UUPP National Advisory Board (NAB) recently began requiring institutions to submit portfolio-like “presentations,” preferably electronic, in place of traditional self-studies.  The UUPP director is currently working with the association to design protocols for campuses that are developing these presentations and for reviewers who are evaluating them.  Another member of the UUPP Leadership Team was elected to the WASC Commission that decides on institutional accreditation and brings experience with UUPP portfolios to those judgments.  UUPP participants have offered extremely well-attended and well-reviewed workshops on electronic institutional portfolios at the last two WASC annual meetings and have been invited to return to the next meeting.

In addition, at least three UUPP institutions plan to use their electronic portfolios as self-studies for upcoming accreditation reviews.  The North Central Association’s Higher Learning Commission has agreed to allow IUPUI to use its electronic portfolio in lieu of a traditional self-study for its 2002 accreditation review.  PSU is negotiating with the Northwest Association of Schools and Colleges for permission to develop its self-study within its institutional portfolio for its accreditation review in 2005.  And CSUS is further developing its portfolio for its next WASC review in 2007.

The usefulness of the portfolios for state review purposes is more difficult to predict.  Discussions at the Hayes meeting in San Jose suggested that state policymakers are supportive of the efforts to examine and improve student learning institution-wide that the portfolios document and represent.  But for accountability purposes, state policymakers noted the need for greater focus on performance indicators and other summative data on effectiveness, as well as more information on student attainment of core learning outcomes, than the portfolios currently provide.  If such data can be developed and incorporated into portfolios in a follow-up project, then electronic institutional portfolios may well provide the kinds of information on student learning that Measuring Up 2000: The State-By-State Report Card on Higher Education found lacking at the state level.

Site Visits

Initially, the annual site visits by IRB members and the UUPP director were envisioned as means for testing the usefulness of the portfolios for accreditation and other forms of quality assurance.  As the project began to unfold, however, it quickly became apparent that portfolio development itself was going to be lengthier and more complex than originally anticipated.   As IRB member Barbara Walvoord points out in her report on the Spring 2001 site visit to UIC, “the universities have found that before they could prepare a portfolio for external audiences, a great deal more internal work was necessary than project initiators had expected.”  Most of the institutions in the project needed to develop not only assessment evidence to include in their portfolios, but also institutional processes for generating that evidence.  Creating and structuring these processes thus became a strong focus of project work for the individual campuses and of discussion at project meetings.

As a result, the site visits took on forms and purposes intended to help the campuses develop assessment programs, to gain campus buy-in for assessment and for the project, and to provide feedback and suggestions from a user standpoint on the evolving portfolios.  For example, site visit agendas typically included meetings with committees whose work might be represented in the portfolios to brainstorm about the possible form of that representation.  Some campuses wanted advice on specific assessment issues. Some scheduled meetings of site visitors with key campus groups and offices, using the visits as occasions to bring greater visibility to their portfolios, to spark conversations about learning and effectiveness, and to gain campus support for the project.  During most visits, site visitors provided detailed suggestions for building and improving the portfolios.  These advising sessions (and subsequent written visit reports) often emphasized issues of navigability, appearance, and language:  the usefulness of the portfolios depends not just on the information they include, but on the ability of users to access and understand this information as easily as possible.

Site visits yielded valuable insights about how electronic institutional portfolios might be used for quality assurance.  We found it essential that campuses prepare summary documents—well in advance of the campus visits—that described the structure and major sections of the portfolios and helped site visitors understand how to navigate the portfolios.  We learned that reviewers typically needed to make multiple visits to the portfolio sites in order to digest the information in them.  We also found that reviewers were easily able to combine the roles of “evaluator” and “consultant.”  While this last finding must be attributed in part to the collegiality that developed among UUPP participants over the course of the project, we believe it is suggestive for accrediting associations that hope to build more supportive relationships with higher education institutions.  It may imply that accreditation could be enhanced by creating mechanisms that encourage more ongoing communication among accrediting associations, review teams, and institutions under review.

The Role of Technology

One intriguing aspect of the UUPP is that the development of the portfolios took a rather different technological path than that originally foreseen.  The project proposal for the UUPP envisions the institutions creating paper portfolios that would eventually migrate to the Web.  Instead, from the outset of the project, all six institutions planned their portfolios as Web sites.  Technology quickly became a central focus of the project as the six campuses together explored the potential of electronic media to enhance institutional effectiveness and the ways we represent our institutions to our publics.   In this sense, it is indeed the case that “in form, content, [and] production,” the portfolios “differ significantly from current kinds of documentation” used for quality assurance.

The project’s technology focus had a profound influence on the processes and outcomes of portfolio development at the six campuses.  On the one hand, campuses struggled with technology issues.  Each campus, for instance, had to grapple with the lack of models for Web-based institutional portfolios.  Determining how to organize and structure the portfolios was especially difficult.  Most campuses went through multiple iterations of their portfolios before arriving at a satisfactory format and design:  PSU’s campus project director notes that “during the three years of the UUPP, PSU’s portfolio was redesigned at least three times….Most of the grant period was spent evolving the concept of the electronic portfolio into something that could be implemented.”  Generally, campuses tended to move from more complex to simpler organizational designs for their portfolios, with most of the portfolios ultimately organized around core components of the campus mission or strategic plan.

On the other hand, the electronic nature of the portfolios and even the struggle to identify useful organizing concepts and principles for them made the process of developing them more powerful and transformative.  Conceptualizing a portfolio that would build on the structuring and linking capabilities of the Web to represent the effectiveness of the entire institution meant that portfolio developers needed to think broadly and integratively about actual structures and linkages on their campuses.  One result was that most campuses ended the project with a significantly expanded pool of faculty members and administrators who had a grasp of the entire institution, of how various parts of the institution related to one other, and of how institutional practices, programs, and initiatives related to mission and strategic priorities.  This in itself was a striking development that led several project institutions, including CSUS, GSU and PSU, to revise their approaches to strategic planning. 

At PSU, this enlarged understanding of the campus was a recurrent theme of site visits over the three years of the project.  In particular, members of the campus’s UUPP Faculty Advisory Committee repeatedly commented on how the work of conceptualizing and building the portfolio led to rich conversations about the institution, its mission, and the meaning of the mission in practice.  By the second site visit, in Spring 2000, committee members already felt that they had gained a more comprehensive understanding of the campus and a clearer sense of how their individual day-to-day work contributed to realizing the mission; the third-year site visit reaffirmed this understanding.  In this sense, creating an electronic institutional portfolio proved to be a compelling approach to faculty and organizational development for this campus.

Similarly, UIC ‘s campus project director reports that the task of developing structure and content specifically for the Web medium created a new context for campus-wide collaboration.  He argues that the UUPP’s focus on developing Web strategies for representing work on learning and institutional effectiveness across the campus required the campus’s project team to take a broad perspective on the institution and to collaborate with people and groups throughout the university to “pursue the goals of assessment through a structure not bound by colleges and units to traditional ways of thinking.”  He goes on to observe that:

The UIC Portfolio Project Team has learned that building a Web site is continuous quality work.  The process is not linear, but exemplary of an assessment loop in the first place, as we ask Who are we? What do we teach?  Whom do we teach?  What goals do we set?  How do we measure those goals?  What processes do we create to assure the quality of education?

As these comments imply, project campuses found that the effort to make virtual structures and linkages coherent and cohesive in the portfolio often  revealed areas where actual institutional structures and linkages were lacking, incomplete, or inconsistent.  Many campus improvement initiatives that emerged from the project began with insights about such structural gaps or weaknesses.  For example, as IUPUI began to build a portion of its portfolio around its six “Principles of Undergraduate Learning” (PULs)—the core skills and abilities that graduates of IUPUI are expected to master—portfolio developers realized that they lacked a complete picture of how the principles were taught, learned, and assessed across the campus’s 22 schools.  This realization led to a study of teaching and learning of the principles across the campus that is now a prominent feature of the IUPUI portfolio.  It also was a significant impetus for a faculty development initiative focused on the PULs, as well as for a new student electronic portfolio program that will provide even more information on student learning of the principles and help the campus and its component schools find the most effective approaches to helping students master these key abilities.

 Lessons Learned 

UUPP participants, ranging from members of campus portfolio development committees to campus administrators and leaders of the project to advisory board members, strongly affirm that the project has been a powerful learning experience.   The preceding discussion has attempted to articulate these lessons in relation to the four project deliverables.  The following summary of those lessons includes some additional comments:

Lesson 1:  Developing an electronic institutional portfolio is a complicated, ongoing  process that can stimulate significant positive changes in institutional culture.

From the beginning of the project, most UUPP participants understood the ambitiousness of our principal task:  to conceptualize and bring to realization a new form of communication about the evolving concepts of institutional and educational effectiveness, with a particular focus on the urban public university context.  But few fully grasped the broad institutional implications of the work ahead.  The portfolios envisioned by the project required evidence and documentation of institutional and educational effectiveness—evidence that, for most campuses in the project, was still relatively scant.  To develop this evidence meant that institutions needed to articulate a collective vision of their purposes, including educational purposes, and to develop ongoing, systematic programs, practices, and support structures for accomplishing those purposes and for determining the extent to which they are fulfilled.  For many higher education institutions, beginning and sustaining such structures and programs requires significant cultural and organizational change that takes more than three years to accomplish.

Some UUPP campus project directors needed to advocate for changes that were not necessarily understood or supported by faculty and administrative leaders and that may not have fit established campus cultures.  Campuses with  well-established institutional research offices and at least the beginnings of a campus-wide assessment program could make faster progress on developing their portfolios.  Campuses with a history of collaboration among units and of open communication about strengths and weaknesses were also at an advantage.  Continuity of support from top administrative leaders was another key factor in building and sustaining momentum for change and portfolio development; at least one institution’s efforts were impeded when new leadership did not support or participate in project work. 

The project thus confirmed, albeit in a new context, accepted wisdom about the difficulty of developing assessment cultures in higher education, the ease with which assessment initiatives can be derailed by lack of ongoing support from key leaders, and the need for patience and sustained effort over years by those working to create such cultures.  That said, institutions also discovered strategies for creating faculty buy-in for assessment and development of evidence of effectiveness.  One strategy used at UMB was to identify “pockets” of assessment activity around issues of teaching and learning that genuinely engaged faculty interest—e.g., how do students learn to think critically or to value diversity?—and then to use those activities as models that might be adapted by other programs or departments.

Another effective strategy, adopted by the UUPP team at UIC, was to develop Web materials for the portfolio on an established writing assessment program and to showcase those materials among various campus groups.  The World Wide Web is still new enough that the opportunity for a department or program’s work to be represented on a university Web site accessible world-wide can be a strong incentive to begin work on assessment and to create new materials for inclusion in the electronic institutional portfolio.

Ultimately, as this report has discussed, the UUPP was a powerful catalyst for the development of assessment efforts and of cultures of evidence.  But for the majority of the six campuses, this work is still (inevitably) at an early stage; continued connections among the campuses would support further development.

Lesson 2:  Productive collaboration among institutions, although challenging to establish, provides essential support for electronic institutional portfolio development.

At early meetings of the UUPP, prospects for fruitful collaboration seemed uncertain:  campuses had varying agendas for and perspectives on the project, varying levels of experience with assessment of student learning, different histories of involvement with national higher education issues and discussions that formed the broader context for the project, and different institutional aspirations emerging from very diverse ideas about the nature of excellence in higher education.  Some campus project leaders worried that the project deliverables suggested a degree of uniformity among the portfolios or the campuses themselves that would impede buy-in for campus work on the project.

Collaborative relationships thus had to be deliberately cultivated.  Multiple meetings, interactions among all project constituencies, and site visits all contributed to building these relationships.  During the first year of the project, extra meetings of the campus directors and project-wide leaders, the groups most central to the progress of project work, helped develop trust, camaraderie, and a sense that all were committed to helping one another succeed.  As these relationships were strengthened, the larger group of project participants benefited as well.  Meetings became forums for collectively studying and critiquing the evolving portfolios, sharing new and creative ideas about presenting material and evidence on the Web, exchanging strategies for gaining campus buy-in for the portfolio and related planning and assessment efforts, learning about assessment and about the national higher education context for the project, and providing support and encouragement for individual campus efforts.

In the end, campus leaders of the UUPP credited the collaboration with other project institutions, project-wide leaders, and advisory board members for much of the progress they made on developing their electronic portfolios and the materials and evidence the portfolios include.  Many ideas originating on one project campus were adapted or adopted by other campuses: examination of the six institutional portfolios reveals the influence of this cross-fertilization in the form of common organizing structures, navigational schemes, types of evidence used, and the use of interactive features, such as the inclusion of discussion boards or “ask a question” areas in several of the portfolio Web sites.  For this group, collaboration—the opportunity to see and discuss other campuses’ developing portfolios—helped make the electronic institutional portfolio concept concrete and was thus especially valuable.

Annual site visits were another aspect of the collaboration that provided opportunities for campuses to receive detailed feedback on their portfolios and related campus work from experts on teaching and learning in higher education, assessment, and documentation of the outcomes of higher education.  These visits were occasions for rich conversations about institutional purposes, strategies for building or enhancing assessment processes, and ways of integrating key offices, programs, and initiatives into the portfolios and the documentation of effectiveness. The opportunity for campus project directors to participate in site visits to other project institutions was cited as extremely valuable to their own campus work by those campus directors who were able to join in site visits.

As a result of the UUPP experience, we suggest that accrediting associations experimenting with electronic institutional portfolios as alternatives to traditional paper-based self-studies consider adapting features of the UUPP collaboration, such as forming cohorts of institutions to work together on portfolio development and arranging periodic site visits to provide formative evaluation of portfolios as they develop.

Lesson 3:  The potential of electronic technologies to change the ways institutions demonstrate their effectiveness and the ways effectiveness is evaluated by stakeholders is attracting broad national interest.

An earlier section of this report discussed how technology and the structure of the Web itself can lead to new ways of conceptualizing and improving institutional structures and practices.  Technology may also change the kinds of evidence and processes we use, both internally and externally, to assess and demonstrate student learning and other core activities of higher education institutions.  For example, the use of multiple media increasingly will allow institutions to show—not merely describe—how teaching and learning actually occur and to present authentic examples of student learning.  This transparency, in combination with the ability to store and structure large amounts and many types of information on a Web site, can help stakeholders seeking to understand what we do in higher education (or at our particular institutions), how we do it, and how well we do it. 

For example, members of accreditation review teams may be able to gain greater understanding and insight into institutions under review prior to traveling to the campus by paying multiple virtual visits to an electronic institutional portfolio.  Virtual visits may reduce the amount of time that evaluation teams will  need to spend on a campus or free up time now spent gathering basic information during campus visits for more extended discussion and feedback on improvement strategies and priorities.  

We still have much to learn about fully exploiting the capabilities of technology to enhance our ability to communicate among ourselves and with stakeholders about effectiveness and quality in higher education.  The portfolios created by the UUPP campuses are, by necessity, initial, incomplete attempts.  But the work of the project, preliminary as it is, has attracted enormous interest among higher education institutions and their stakeholders.  The discussions at the meeting on “Urban University Portfolios:  Performance and Prospects” provide some indication of this interest among various stakeholders.  Members of the UUPP National Advisory Board, who similarly represent a range of stakeholder groups, have been keenly interested participants in and advisors to the project.  Several regional accrediting associations, perhaps influenced in part by the UUPP, are now experimenting with the use of electronic institutional portfolios in the self-study process; directors and staff of these associations, including the Western Association, the North Central Association, and the New England Association, have sought out advice from UUPP participants, while five out of the six regional accrediting bodies have included presentations and workshops on the UUPP portfolios in their annual conference programs. 

During the last year-and-a-half of the project and since its formal conclusion, our presentations at both national and international conferences on higher education have drawn standing-room-only crowds.  We have been particularly pleased by the level of international interest and enthusiasm at conferences in Australia, Germany, Scotland, and Portugal.  Higher educators and stakeholders in other countries, like those in the U.S., are intrigued by the potential of technology to contribute to more meaningful approaches to demonstrating and improving educational effectiveness and see the work of the UUPP as a model.  Similarly, our longer workshop sessions, aimed at helping participants begin portfolios on their own campuses, have been fully subscribed and have led to several campus invitations for project-wide and campus leaders.   A recent AAHE volume on electronic portfolios that features the work of the project quickly sold out when it was released at AAHE’S June 2001 Assessment Conference; a special issue of Metropolitan Universities on the UUPP is forthcoming in Summer 2002. 

Lesson 4:  Development and maintenance of electronic institutional portfolios demands significant resource commitments.

During the last year of the UUPP, a study of the financial, technological, and human resources required to develop and maintain electronic institutional portfolios was conducted by Edutech International.  Edutech consultants, along with the UUPP Technology Associate, visited each project campus, conducting extensive discussions with campus leaders and participants.  A detailed report, with recommendations, was prepared for each individual campus in the project.  Project-wide findings are summarized in the overall report, which is attached.       

Perhaps predictably, given the different directions and emphases of the project at the six campuses, the Edutech consultants found that the level of necessary resource commitment varies, depending on an institution’s pre-existing technology infrastructures and staff skills, the institution’s purposes in creating its portfolio, and the technological ambitiousness of the portfolio.  Most UUPP campuses thought that Edutech may have overestimated staffing needs.  Nonetheless, electronic institutional portfolios do require a substantial financial commitment from an institution.  In the future, it will be interesting to compare the cost of a long-term, ongoing portfolio project with the cost of a once-every-ten-years comprehensive accreditation self-study or of preparation for state reporting for public institutions.  It may well be that electronic institutional portfolios are the more cost-effective and useful alternative in the long run.

Plans for the Future

Many participants in the UUPP are eager to continue to develop models for electronic institutional portfolios, to explore new ways of presenting information and evidence for effectiveness on the Web, and to test uses of the portfolios with a range of stakeholders and for a variety of purposes.  Specific plans include these: 

• At least five of the six institutions in the project plan to continue to develop their portfolios.  At the sixth campus, UMB, a new chancellor, who had not yet arrived when the project concluded, will decide whether or not to continue the portfolio and related initiatives.  Some institutions, like GSU, may use the portfolio primarily for internal purposes of planning and program review.  At least two institutions, IUPUI and PSU, are developing their portfolios as self-studies for upcoming comprehensive accreditation reviews.

• Project participants will continue to disseminate the work of the project and the ongoing development of their portfolios.  Experiments by several project institutions with using portfolios for accreditation promise to provide fertile materials for continued dissemination.  As WASC implements its new requirement for portfolio-like “presentations” in lieu of traditional self-studies for accreditation, we expect that institutions in the WASC region will continue to be interested in our approaches to developing portfolios for accreditation. 

• IUPUI and AAHE are seeking funding for a follow-up project that would build on the work of the UUPP and extend it to other types of institutions.  We are especially interested in exploring the ways in which various types of electronic portfolios (student, faculty, department/program, institutional) might be used in combination to provide evidence of student learning and institutional effectiveness for accreditation, state reporting, and other purposes.