This report summarizes work on the Urban Universities Portfolio Project (UUPP) from July 1999 through June 2000.  The report focuses on campus and project-wide progress toward key project deliverables, major project activities and outcomes over the year, important lessons learned, and plans for the third and final year of the project.  Additional details can be found in the individual campus reports and other materials in the attached appendices.

Major Developments and Current Status

Key Focuses

This year’s project work had several key focuses:

•         Beginning or further developing the individual campus portfolios, generally in connection with concurrent campus initiatives on learning outcomes, assessment, strategic planning or institutional effectiveness.
  

•         As a whole project, discussing, developing and exchanging ideas about assessing and documenting student learning activities and outcomes in the portfolios. These discussions were major agenda items at three project-wide meetings and one campus directors’ meeting over the course of the year.
  

•         Refining our ideas about the common characteristics and mission components of urban public universities, with a strong focus on engagement with the urban community as an important defining characteristic.
  

•         Beginning to consider and discuss how to document the impact of our campuses’ civic engagement on our urban communities.
  

•         Conducting a first round of experimental team site visits to each UUPP campus to explore how portfolios can provide new approaches to accreditation and other forms of institutional evaluation.
  

•         Expanding the involvement of both internal and external stakeholders at the campus and project-wide levels.
  

•         Collaborating with related Pew initiatives, especially the Western Association of Schools and Colleges (WASC) and North Central Association projects, and the National Survey of Student Engagement (NSSE).
  

•         Disseminating the outcomes of the project widely through publications and conference presentations.

Important Advances

Our work on these areas of focus led to significant progress and a number of important advances.  Each of the advances listed below is discussed in greater detail in later sections of this report.

• Each project campus now has an initial portfolio on the World Wide Web, including an overall framework and individual sections that are well developed.  (URLs for the individual portfolios are listed in Appendix B.) While the individual portfolios vary in structure and intended uses, they are recognizably part of the same “genre”; thus, we have a much more fully delineated idea of what an electronic institutional portfolio “looks like” than we did a year ago, as well as a working definition of “institutional portfolio.”

• The project has been a catalyst for significant assessment and improvement initiatives on every participating campus.

• Involvement of campus faculty members and administrators has expanded  on every project campus.

• Because some campuses began constructing their on-line portfolios earlier than others, and the portfolios are currently at various points in their evolution, we now have a sense of the “stages of development” of an institutional portfolio, of the issues that come up during the early stages, and of what needs to be in place on a campus to begin an initiative like this one.   We also can articulate more clearly than a year ago the  benefits of developing an electronic institutional portfolio.

• The project’s Technology Development Associate has identified an outside consultant to assess technological and staff capacities and needs for developing and maintaining institutional portfolios.  The consultant’s findings will contribute to our understanding of the infrastructure necessary to support ongoing work on an electronic institutional portfolio.

• As a result of ongoing discussion within the project and of our collective participation in the National Survey of Student Engagement, we have refined our ideas about what differentiates urban public universities from other types of institutions and moved closer to developing a set of “urban” indicators of effectiveness.

• Insights gained from feedback from external stakeholders have begun to be incorporated into the portfolios.

• Significant external stakeholders, especially our regional accrediting associations, are aware of and interested in the project.

• We are analyzing the results of the initial round of experimental site visits, both virtual and physical.  The results of our analysis will help us to plan for next year’s more formal visits, and implement any needed changes to the structures and procedures for the visits.
  

Progress on Key Project Deliverables

The UUPP grant agreement with Pew lists four key deliverables for the project.  Progress on each of these deliverables is discussed in this section.

1.  The participating institutions will have a body of evidence that they can use for   internal improvement.

Portfolio Development

All six universities participating in the UUPP now have initial versions of their portfolios on the web, including those campuses that had not yet begun constructing portfolios at the time of the last report.  Most of the portfolios are organized around a set of topical themes, reflecting major categories of institutional work and priorities, with student learning (and its assessment) and civic engagement as key sections.  Some of the portfolios are taking advantage of the web environment by including a second, concurrent organizing scheme, often focused on the concerns of potential groups of portfolio users:  students, community members, accreditors, state government, and other stakeholders.  For example, visitors to the Portland State University (PSU) or Indiana University Purdue University Indianapolis (IUPUI) portfolio sites will be able to explore the universities’ work with their urban communities—a topical area—in depth, or follow “guided tours” that provide information on topics targeted to specific stakeholder groups.

Each UUPP portfolio includes, or will include, an array of materials and evidence that fall into several common categories, providing some commonality across the portfolios that gives them a similar look and feel.  Among these materials are:  authentic examples and samples of student work (sometimes accompanied by audio or video commentary from students); assessment data and reports; survey results and reports; statistical information, such as statistical portraits of student bodies; and analysis and reflection intended to frame and provide context and coherence to the other materials.  PSU and IUPUI also plan to link representative student portfolios to the institutional portfolio, as illustrations of what and how students learn.  To bring a sense of unity and purpose to all of this information, some campuses are conceptualizing the portfolio as a “story” or set of “narratives” around several common themes, or as a “persuasive essay.”  These analogies seem to be helpful for keeping the portfolios focused on a limited set of clear issues.

At California State University, Sacramento (CSUS), the portfolio project has led to an interesting variation and potential adjunct to the institutional portfolio:  a prototype for an electronic  department/program portfolio focused on teaching and student learning.  The main CSUS institutional portfolio is organized around the campus’s strategic planning themes and focuses on campus work to assess and improve its effectiveness in the various thematic areas, which include teaching and learning.  More detailed information on student learning, including learning outcomes and assessment within programs, and samples of student work, will appear in the program portfolios, which will be linked to the main portfolio.

The program portfolio is currently being piloted in several CSUS departments, as is a university-wide program portfolio on writing, a core baccalaureate learning outcome.  The CSUS prototype has the potential to be a national model for a new type of unit-level electronic portfolio that shifts the focus of departmental program reviews to student learning outcomes.  Already, Georgia State University (GSU) is adapting the CSUS prototype for its own use as an adjunct to the GSU institutional portfolio.  (The CSUS prototype can be viewed at http://www.csus.edu/soc/portfolio/.)

Portfolio Audiences

As the portfolios have continued to develop, UUPP participants have become increasingly aware of the flexibility and adaptability of the portfolios for different purposes and audiences.  Among the six portfolios, some are intended for a wide array of purposes and stakeholder groups, while others are targeted to very specific uses and audiences.  The flexibility of electronic portfolios, however, makes it possible for a campus to incorporate material and pathways for additional audiences in the future, should the need arise.

For example, the CSUS and GSU portfolios are closely linked to ongoing campus strategic planning and institutional effectiveness processes, with internal stakeholders and accrediting organizations as primary audiences.  The portfolios are, in effect, sites that support and document planning, work under way to implement plans, efforts to measure and assess progress on planning priorities, and results of these efforts.  

IUPUI and PSU’s portfolios are perhaps the two with the broadest sets of intended audiences and uses.  Both campuses are addressing their portfolios to prospective students, community members, businesses and employers, state government, and accreditors, as well as to internal stakeholders.  Crafting portfolios that “speak” to such diverse groups of stakeholders, while communicating accountability to each group, has been an important focus of portfolio work for these campuses, requiring special attention to language and site navigation concerns.  Both campuses are constructing special pathways through their portfolios for members of particular stakeholder groups. 

The University of Illinois at Chicago (UIC) and the University of Massachusetts Boston (UMB) have chosen a middle path between the highly targeted CSUS and GSU portfolios and the very broadly inclusive IUPUI and PSU portfolios.  UIC’s portfolio is intended for internal users, students and their families, and the many organizations in the Chicago area involved in partnerships with UIC.  UMB’s portfolio focuses on informing internal audiences about standards and practices related to student learning, and on providing evidence of student learning and community contributions to accrediting bodies and other agencies seeking evidence of effectiveness.

Campus Impact

All of the campuses see internal improvement as a major purpose of the UUPP and the institutional portfolio.  Wide campus involvement in the project is key to achieving this purpose, and virtually every project campus undertook successful efforts to expand such involvement over the past year, particularly the three campuses that had not established broad participation during Year 1.

Most UUPP campuses report that the portfolio has enhanced their strategic planning efforts by bringing together key information and documents in a coherent way, enabling relationships among various activities to be more clearly understood, and making unmet needs and areas for improvement more readily apparent.  PSU, which has lacked a formal strategic planning process, reports that it is now developing one as a result of the project.  GSU has used its participation in the UUPP as an occasion to develop a new strategic plan that takes advantage of the web environment to facilitate campus-wide interaction and participation in the development of the plan.

Similarly, other UUPP campuses have launched a variety of improvement efforts to address needs identified in the course of developing their portfolios—or preparing to develop them—especially in the areas of learning outcomes and assessment of learning.  Catalyzed by participation in the project, UIC has adopted a set of learning outcomes for the baccalaureate degree and is beginning the work of incorporating the outcomes into curricula and developing appropriate assessment strategies.  CSUS is also working on defining learning outcomes; having completed a survey of internal and external stakeholders on key outcomes, the campus is beginning a two-year pilot assessment project that includes six fundamental learning goals, and is planning a follow-up survey of a larger group of community members next year.

At IUPUI, work on the institutional portfolio has spurred a campus-wide initiative to study how the campus’s six “Principles of Undergraduate Learning” are incorporated into curricula and assessed campus-wide.  The results of the study will be included in the portfolio, which will serve as a portion of IUPUI’s self-study for its next accreditation review in 2002, and will guide planning for future curriculum and faculty development to ensure effective implementation of the principles.   UMB and GSU also report new campus initiatives on assessment and documentation of learning outcomes as a result of work on the UUPP.

Work on the project continues to suggest that institutional portfolios have significant implications and potential as vehicles for faculty and organizational development.  A discussion with the Faculty Advisory Committee at Portland State, during this year’s site visit, served as dramatic illustration of this potential.  Committee members commented that work on the portfolio has broadened their perspective on the institution beyond their own departments, helped them see the institution as a coherent whole, and showed them that the institution’s mission is “real.”  They noted that they were surprised by how much of the mission is “really” being carried out via various campus initiatives, and by the way these activities complemented one another in supporting the mission and giving it depth and breadth.  One said that the portfolio work helped her “bridge the gap” between the institution’s mission and her everyday activities—that is, to see how her individual work fit into the overall work of the institution.

Clearly, for this group of faculty, collaborating on development of the institutional portfolio has enhanced their understanding of how the university works to achieve its mission, and strengthened their commitment to helping the university carry out that mission as effectively as possible.  The question for Portland State, as for other institutions in the project, is how to expand internal involvement and commitment even further, both to institutionalize the portfolio, so that portfolio development will continue beyond the funded period of the project, and so that even more internal stakeholders enhance their understanding of the university’s mission and their commitment to effectiveness within that mission.

A Proposed Definition

With all of the above aspects of our portfolio work in mind, we have developed the following working definition of “institutional portfolio”:  "a focused selection of authentic work, data and analysis that demonstrates accountability and serves as a vehicle for institutional reflection, learning and improvement."

2.  The institutions will have improved their capacity for public accountability.

Internal Improvement and Public Accountability

Developing a “body of evidence…for internal improvement,” the deliverable discussed in the above section, is an important step in moving toward this second deliverable, improving “capacity for public accountability.”  As UUPP institutions gather and display evidence of effectiveness, identify areas needing improvement, and take steps to implement improvement, their capacity for public accountability is also enhanced; a broadly shared commitment to “internal improvement” and a “capacity for public accountability” go hand in hand.   Mary Breslin of the North Central Association emphasized this point during the IUPUI site visit.  When asked what North Central expected in the portfolio, she responded that the portfolio should focus on the improvement needs identified by the institution itself:  “If it works for IUPUI, it’ll work for North Central.”

Involvement of External Stakeholders

In addition, as noted in the above section on Portfolio Audiences, the portfolios are intended for use by specific groups of internal and external stakeholders.  During the first year of the project, UUPP campuses had limited involvement of external stakeholders in the project; most felt that the portfolios needed to have substantial content for productive discussions with external stakeholders to take place.  In this past year, some of these discussions have begun, through public presentations to accrediting organizations and state boards, through focus groups, and through the spring site visits.  For example, Portland State presented its portfolio to the Oregon State Board of Higher Education; IUPUI and UIC presented theirs at the annual meeting of the North Central Association; and CSUS discussed its project work at WASC’s annual meeting.

Portland State has used focus groups of external stakeholders to help the project team maintain “a focus on audiences and questions of interest” and to “link portfolio content to assessment and accountability concerns,” and plans to expand involvement of these focus groups next year.  Two campuses, CSUS and IUPUI, included representatives of key stakeholder groups in their site visit teams this year:  a WASC staff member joined the CSUS site visit team; and the IUPUI team included a representative from the North Central Association, the Executive Director of the Indianapolis Chamber of Commerce, an Assistant Deputy Mayor for Neighborhoods, and two students.

IUPUI’s campus director notes in her annual progress report that including these stakeholders on the team made the site visit a “powerful reality check” on the portfolio’s usefulness for public accountability.  Comments by these site visit team members highlighted issues of terminology and site navigation and will help guide the development of the next iteration of the portfolio.  An important lesson of this site visit was that the portfolios will be useful for external accountability only if they are accessible to external stakeholder audiences in language and ease of use.

Civic Engagement

One area of public accountability where the UUPP may be able to make a valuable contribution is in assessing the impact on our urban communities of civic engagement, a key part of the mission of urban public universities.  UUPP institutions began working on this issue during this past year and have identified it as a priority for Year 3 of the project.  Additional details about our work on civic engagement are provided below, under Deliverable 3.

Functional Needs Assessment

When a campus uses an electronic institutional portfolio to demonstrate public accountability, “capacity for public accountability” must include the technological infrastructure and staff skills needed to develop and maintain the portfolio.  The UUPP requires participating institutions to align technical, analytical, and graphic design resources in ways that most have never done before.  Each campus must address issues related to emerging computing platforms, staff skills, and the development of new working relationships among campus units.  With new web development and computer-based assessment tools appearing at a rapid rate, development choices have become increasingly complex. 

To address these issues, the UUPP’s Technology Development Associate developed and circulated an RFP seeking an outside consultant to assess UUPP campuses’ capacities and available resources for designing, developing and maintaining electronic institutional portfolios.  This fall, the consultant selected will develop a “functional needs assessment” report for each campus and for the project as a whole.  The project-wide report will identify areas of commonality among the campuses that may allow some or all of them to develop common tools or invest in common solutions, and will include recommendations for the best use of UUPP financial resources for technology support.  The results of the functional needs assessment will contribute to the project’s understanding of the human and technological infrastructure necessary to support ongoing work on an electronic institutional portfolio.        

3.  The project will produce a core set of indicators of effectiveness that, although they will be designed specifically for one sector of universities, can be used as a template by others.

Indicator Development

Development of indicators is proceeding similarly to the way portfolio development in general is proceeding.   Each UUPP campus has created its own process for building its institutional portfolio and its own design for the portfolio; these processes and designs grow out of the particular culture and needs of the campus, its reasons for participating in the project, and the project’s relationship with other campus initiatives and improvement efforts.  At the same time, an examination of the six portfolios reveals considerable cross-fertilization among the six campuses, as well as common themes and modes of documentation that reflect the influence of project-wide discussion and collaboration.

Similarly, each campus is working with its existing indicators or creating indicators appropriate to its particular goals and circumstances.  Concurrently, cross-fertilization resulting from project-wide discussion is influencing the selection and use of indicators for the individual portfolios.  The next step will be to determine a set of core indicators through analysis of the six portfolios and to document these indicators on a UUPP “meta-site” that brings together common elements of all six portfolios and presents the collective results of the project.  (See the section on “Plans for Next Year” for additional discussion of the meta-site.)  At this point, however, some preliminary conclusions about core indicators can be reached.

Urban Indicators

During the first year of the UUPP, participants identified four defining characteristics of urban public universities:  very diverse student bodies, with high proportions of non-traditional and transfer students; strong emphasis on programs geared to professional preparation and to interdisciplinary study of urban issues; engagement with the city in teaching, research and service; and commitment to access.  Using the four characteristics and the current UUPP portfolios, project staff have generated a list of six categories as a possible framework for our “urban” indicators and documentation of work on the urban mission:  access and support; student learning in the urban setting; diversity and pluralism; civic engagement; urban relevance of programs and scholarship; and urban universities as intellectual and cultural resources for the city.  Subject to revision at the August 2000 UUPP meeting, the six categories and their elaboration in the UUPP portfolios will be addressed in detail on the UUPP meta-site.

Here are examples of how several of the categories are addressed in the current portfolios:

Diversity and pluralism; access and support; student learning in the urban setting.  All of the campuses include statistical portraits of their student bodies to show just how diverse they are, as well as to track trends.  Increasing student diversity and meeting the needs of a diverse student body through academic and student support programs are explicit or implicit indicators used by at least three UUPP campuses, CSUS, UIC and UMB, to demonstrate their effectiveness in carrying out their urban missions.  These campuses’ portfolios suggest various measures for these indicators, which relate to both “access and support” and to “diversity and pluralism”.

For example, UIC provides an array of evidence to show that it is effective in maintaining a diverse student body and providing needed support for those students.  Evidence and measures cited in the UIC portfolio include statistics on minority representation at the undergraduate and graduate levels, the percentage of medical school graduates who are minorities, statistics showing increased diversity over time, national recognition and awards related to diversity, diversity events on campus, and information about student support programs.

In addition, the UUPP group’s participation in the National Survey of Student Engagement allows us to compare diversity-related learning experiences of students at urban universities with those of students at other colleges and universities.  For example, sophomores and seniors at urban public universities report higher gains in “understanding people of other racial and ethnic backgrounds” than their peers at other large public research universities.  This finding might fall under “diversity and pluralism” and/or under “student learning in the urban setting.”

Civic Engagement.   All of the portfolios cite engagement with the urban community as a key focus of institutional work, and all include examples of urban partnerships in a range of categories, such as education, health, economic development, arts and culture, and environmental issues.  Over this past year, project participants have come to realize that enumerating and describing these partnerships indicates commitment to positive engagement with our communities, but is not sufficient to demonstrate the impact of this commitment in contributing to the quality of life in our cities.  We have thus begun discussing as a project how we might assess and document our “civic effectiveness” in the portfolios, and plan to make this an important focus for the coming year.

Several project campuses have the beginnings of such documentation in their portfolios.  For example, Portland State’s work on assessing the impact of service learning on both students and community partners suggests several possible indicators of effectiveness of student work in the community, including impact on the community partner’s capacity and perceived value of the services received.  CSUS’s work in this area may also prove helpful for other institutions in the project.  In its strategic plan, CSUS categorizes its work in the community according to the kinds of issues the work addresses—human and social services, culture, economic development and so on—and then lists indicators of effectiveness in each category.  This approach may yield more meaningful information for accountability and improvement than would indicators that cut across all areas of engagement.  

The UUPP’s work on indicators of civic engagement has the potential to influence other institutions wishing to assess their effectiveness in this area.  For example, the organizer of the most recent Urban 13/21 meeting, an associate provost at Cleveland State, recently commented that the Urban 13/21 group has taken up this issue, in part as a result of the UUPP’s work.

Student Learning

Documentation of student learning in the portfolios reflects the current status of assessment at the various UUPP campuses.  Most, though not all, UUPP campuses began the project at an early stage of developing systematic institution-wide assessment of student learning; indeed, several joined the project for the purpose of using it to help develop the institution’s assessment program.  Tracking their progress and the campuses’ strategies for implementing assessment programs has taught us some valuable lessons that confirm the difficulty of developing an assessment culture at a large, complex campus.  At the same time, as reported in the “Campus Impact” section above, every campus in the project has made significant progress, catalyzed by the UUPP, in implementing campus-wide assessment or in improving existing assessment approaches. 

All six portfolios include major sections on student learning outcomes in their frameworks.  These sections are at various stages of development, and include a range of different types of documentation of learning.  CSUS, for example, makes extensive use of survey results to document progress on student learning and other key themes of its strategic plan.  UMB uses results of its participation in the NSSE, linking specific NSSE questions to core baccalaureate learning outcomes, and reporting summaries of relevant NSSE responses for each outcome as indirect evidence of student learning of the outcome.

The most fully developed learning outcome currently is the Writing section of the IUPUI portfolio.  This section provides information on required sequences of writing courses and enrollment patterns for these courses, portfolio assessment of writing in the courses, grading rubrics and grade distribution patterns, and criteria for passing each course.  Other documentation includes information on writing placement testing procedures and rubrics, and samples of student writing at different placement levels.  Results of alumni surveys and of the Fall ’99 pilot of the NSSE provide a sense of how recent graduates and current students evaluate their own progress in writing at IUPUI. 

A novel feature of this section of the IUPUI portfolio is a study by an honors student of undergraduate writing at IUPUI. Using student writing samples submitted for inclusion in the IUPUI portfolio, supplemented by interviews with students and faculty, this student investigated and reported on how undergraduate students learn to write, what faculty in various disciplines value in student writing, and how the writing-to-learn movement has influenced writing instruction at IUPUI.  Finally, this section of the IUPUI portfolio describes faculty development programs on student writing, results of evaluations of those programs and plans to improve them, and reports on several new assessment initiatives under way that will provide additional evidence on student writing.  Sample syllabi and assignments from the core writing courses, writing assessment results from individual program reviews, and additional student writing samples will be added to the site over the next year.

Portland State has just begun efforts to assess for specific learning outcomes, but has done extensive assessment of service learning, a “good practice.”  Persuasive information on the effectiveness of this practice is reported in the form of case studies of three recent Senior Capstone courses that incorporate service learning; documentation includes course syllabi, examples of student work, assessment tools, and video and audio clips of students’ reflections on what they have learned.  While the student comments are not presented as evidence for specific outcomes, they are suggestive of learning in such areas as understanding diversity, developing values, and civic responsibility.  The beginnings of outcomes-based assessment efforts at PSU are reported in the portfolio’s section on its Freshman Inquiry program and in the report of the Commission on Campus Climate and Life.

Given that most UUPP campuses are still developing their approaches to assessing and documenting learning, we devoted much of our meeting time over the past year to discussing and planning assessment strategies for such learning outcomes as writing, pluralism and diversity, and critical thinking.  We also addressed ways of documenting learning outcomes in the early stages of an assessment program.  For example, UIC and UMB are planning to highlight exemplary assessment efforts and results by individual faculty or departments, and to include “stories” of individual students’ mastery of key outcomes, along with student work samples illustrating their learning.  Most of the portfolios also include reports and other documentation on the development and progress of their assessment initiatives.  We hope these models will be useful to the many campuses that, as yet, lack full-fledged assessment programs, but are required to document student learning and improvement efforts for accreditation and other purposes.

Now that organizing frameworks are in place for each of the six portfolios, and now that the project has some collective sense of the features and components of electronic institutional portfolios, the campuses are poised to make further progress on fleshing out the documentation of student learning in the portfolios over the next year.  Most of the six campuses have student writing assessment programs in place or in the works; thus, we expect student writing to be the most fully developed and documented outcome across the six portfolios.  In addition, IUPUI, PSU and UMB have indicated plans to document diversity as a learning outcome.  IUPUI plans to document all six of its Principles of Undergraduate Learning through its campus-wide initiative to study implementation of the Principles of Undergraduate Learning, described in the Campus Impact section above.

The National Survey of Student Engagement

To continue developing our ideas about how urban public universities differ from other higher education institutions, and to help shape our ideas about the features of student learning at urban public universities, five UUPP campuses (all but CSUS) participated in the Fall ’99 pilot of the National Survey of Student Engagement.  Survey results from UUPP campuses, along with several other urban campuses, were analyzed as a group, and suggest some possible indicators of “urban learning” that will be taken up in the third year of the project.  Please see Appendix D for a full report on the results of this analysis.

Briefly, a comparison of our group results with results from other large public universities confirms that students at urban public universities have very different educational experiences:

• Our students often combine their education with concurrent off-campus work and family responsibilities; according to the NSSE, they spend more time outside class attending to these responsibilities and less time interacting with fellow students and faculty than their counterparts at other large public universities.  They also report higher rates of in-class participation.  For them, the classroom is the primary point of contact with the university.  These findings suggest that pedagogies that encourage active student participation and that help create a sense of community in the classroom may be especially effective for students at urban public universities.

• Fewer students at urban universities than at other large public universities have participated in field experiences or community service as part of their education.  This finding seems curious in light of the special efforts by many UUPP and other urban campuses to incorporate such experiences into academic programs.  The finding may call for increased efforts by urban campuses to emphasize such experiences, but may also reflect the realities of urban students’ busier lives and their existing work and family commitments.

• Particularly at the sophomore level, urban university students report larger gains in core learning outcomes, such as writing, speaking and critical thinking, than students at other large public universities.  These gains indicate an important “value-added” dimension of higher education at urban universities, where higher proportions of students are under-prepared for college work.  These results also suggest that the UUPP’s focus on learning outcomes may be especially appropriate for demonstrating our effectiveness.

• Overall, urban university students report less satisfaction with their educational experiences than students at other large public universities.  This finding may, again, reflect the more hectic lives of urban students, but also underscores the need for us to ensure that support services are readily available to students, that the campus environment is comfortable and welcoming, and that we cultivate the development of a sense of community in all aspects of our work with students.

The Urban University Statistical Portrait Project

Since February 1998, representatives of urban and metropolitan public universities nation-wide have engaged in a dialogue about how to improve measurement of the characteristics and contributions of their universities. These discussions have occurred at meetings of the Urban 13/21 Academic Officers and the Coalition of Urban and Metropolitan Universities, as well as the UUPP.   Using the emerging commonalities identified through the UUPP as a conceptual framework, proposals were developed and presented to the leadership of both the Coalition of Urban and Metropolitan Universities and the Urban 13/21 Academic Officers to support a targeted effort to assess the measurable characteristics of urban public universities.  Both groups have pledged funding support.

The Urban University Statistical Portrait Project will now proceed based on the urban university characteristics that have emerged from the UUPP.  The groups advising the Stat Portrait Project will provide a broad base of validation for the emerging core concepts, such as access, diversity, and civic engagement. The staff for the Stat Portrait Project will overlap significantly with staff for the UUPP, ensuring that these two efforts remain closely integrated.

In short, indicator development is progressing on a number of fronts, using a variety of approaches—on individual UUPP campuses, through project-wide discussion, and through collaboration with related initiatives like the NSSE and the Urban Stat Portrait Project.  Through all of these efforts, we expect to make further progress over the next year in developing indicators of both “urban-ness” and student learning.

4.  Through the example of the institutional portfolio and external review, the project will yield a more appropriate approach to quality assurance and should improve both state accreditation and state review practices.  In form, content, production and purpose, the portfolios and the process of reviewing them will differ significantly from current kinds of documentation used in program evaluation and institutional accreditation.

Site Visits

The UUPP is addressing this deliverable primarily through two series of site visits to project campuses.  The first series of more informal and experimental visits took place this spring; analysis of reports on the organization and effectiveness of these visits will guide development of procedures and formats for the second series of more formal visits, to be held next year. 

This year’s visits had several purposes:  to determine whether a combination of virtual visits to the institutional portfolio and a physical visit to the campus might be an effective approach to evaluating key aspects of institutional work, and to gain some sense of how virtual visits prior to the physical visit might change the focuses and procedures for the physical visit; to develop some insights into whether visitors can effectively combine “consultant” and “evaluator” roles; and to help UUPP campuses identify priorities and plan the work of the third and final year of the project.

Organization of the Visits 

Written guidelines for this year’s visits are included here in Appendix E.  Each campus was assigned a visit team that included its two IRB partners, a member of the UUPP leadership team, and a representative of another UUPP campus.  Campuses had the option of adding external constituents to the team, and two campuses, CSUS and IUPUI, exercised this option.  Visitors were asked to focus on four areas:  a campus statistical portrait; evidence of the influence of urban-ness on the campus’s practices and environment; outcomes of student learning verified through evidence; and effectiveness of the ways this information is conveyed to multiple publics.

Each campus also developed its own set of questions for the visitors to consider; these generally focused on the effectiveness of the portfolios in conveying key information and presenting evidence, although a few campuses included questions on issues of assessment and civic engagement not specifically focused on the portfolio.  These appear to be appropriate focuses, given the stages of development of the various portfolios as well as the status of assessment on the campuses.

Visit Reports

Several reports were generated from each visit:  “process reports” from each visitor and from the campus project director that summarize the effectiveness of the components of the visit for addressing the project-wide and campus-specific focus areas; and “campus reports” from the IRB representatives that summarize the team’s conclusions about the focus areas themselves.  These latter reports (also included in Appendix E) concentrate on providing guidance to the campuses on their evolving assessment programs and on further development of the portfolios.  A full analysis of all of the reports will be carried out by members of the leadership team this summer and fall, and used in planning next year’s visits, which are conceived as more formal experimental site visits that will influence current national work on redesigning accreditation.

Full analysis of the March-June 2000 visits is just beginning; an evaluation of the usefulness of both the virtual and physical visits and of the portfolios for accreditation and other institutional evaluation purposes will be included as a major segment of next year’s annual report.  So far, reports from the campuses suggest that both the consulting and evaluation components of this year’s visits were useful to them in helping to advance portfolio development and, in some cases, campus assessment initiatives.  Most of the campuses also took advantage of the opportunity of a visit by a national team of assessment and portfolio “experts” to get key people and groups on campus more involved in the project, and to begin or continue to integrate the project with related campus initiatives.  In the case of CSUS and IUPUI, the visits were also an opportunity to inform important external stakeholders about the campus and project, and to encourage their involvement.  Expanded internal and external involvement in developing the portfolios will be crucial to institutionalizing portfolio work beyond the initial grant project.

Other Important Accomplishments

Dissemination of the Project

The work and results of the UUPP were disseminated widely this year, with formal presentations on the project at 16 regional, national and international conferences, several additional presentations to statewide groups in California, Indiana and Oregon, and several publications.  Presentation venues included the annual meetings of three regional accrediting associations, Middle States, North Central and WASC. Presentations at other regional accreditation conferences are planned for next year.  The UUPP Director and Institutional Research Director reported on the project at both meetings of the Urban 13 this year.  Presentations were also made at AAHE’s National Conference, Assessment Conference, and Summer Academy, and at the Association for Institutional Research and the Pacific Northwest Association for Institutional Research.  One presentation, by the Campus Director at Portland State, was voted “best presentation” at the Pacific Northwest Association for Institutional Research Conference, and was repeated as an invited presentation at the national conference of the Association for Institutional Research.  A list of presentations on the project is attached as Appendix C.

The UUPP Director continued to write monthly reports on the project for the “News” section of the AAHE Bulletin; reports appeared in the proceedings of the Middle States and North Central conferences; and articles featuring the project were published in Metropolitan Universities and the Assessment and Accountability Forum.  Appendix C also includes a list of the past year’s publications on the project.

In addition, the UUPP Technology Associate, along with other project staff, continued to develop the UUPP web site (which can be accessed at http://www.imir.iupui.edu/portfolio/).  Since its release in August 1998, the site has hosted over 8,400 visitors, who spent an average of 14 minutes at the site.  Visitors from academic domains (.edu) outnumbered all others, with users from commercial domains (.com) close behind.  While visitors from the United States predominate, users from seventeen other nations also accessed pages from the site. 

The project will be disseminated even more widely next year.  Four dissemination venues are already committed for the summer and fall.  Plans include presentations at the Twelfth International Conference on Assessing Quality in Higher Education in Melbourne, Australia; the National Council of Teachers of English Global Conversations Conference in Utrecht, the Netherlands; the annual conference of the National Association of State Universities and Land Grant Colleges; and the annual meeting of the Southern Association of Colleges and Schools.  AAHE’s upcoming monograph on electronic portfolios in higher education will include a section on institutional portfolios, edited by the UUPP director and with contributed sections by other project participants, that will prominently feature the project and its lessons.

Involvement of External Stakeholders

At the project-wide level, involvement of external stakeholders continued to be a focus of attention this year.  Representatives of the Carnegie Foundation for the Advancement of Teaching, and of WASC, North Central and Middle States participated in our August 1999 all-project meeting.  We have continued to work closely with these three regional accrediting associations and other groups involved in reforming accreditation.  Following our presentation at the North Central Association annual conference, the director of North Central’s Academic Quality Improvement Project initiated discussions with project leaders on the feasibility of using institutional portfolios as documentation of campus work on the project.

For next year, we have already issued invitations to a small meeting in October of UUPP CEO’s and provosts with high-level representatives of key stakeholder groups, including the education media, state government, SHEEO’s, accrediting organizations, Carnegie, and others.   The purpose of the meeting is to encourage these educational policymakers and influencers of policy to determine how urban public universities can be as responsive as possible to the educational needs of their students and regions.  In the process, we hope to increase their understanding of urban institutions’ particular missions, roles, contributions and challenges.   The meeting will be facilitated by the Aspen Institute to help us ensure concrete results.

Project Lessons

Benefits of Developing an Institutional Portfolio

We continue to be struck by the value to a campus of developing an institutional portfolio. The process can be a powerful vehicle for internal improvement and organizational development, leading to more widespread understanding of institutional mission and deeper commitment to educating  students as effectively as possible.  This impact was clearly articulated by the Faculty Advisory Committee at Portland State in the conversation cited in the Campus Impact section above.   Similar insights are emerging at other UUPP campuses.  IUPUI’s director, for example, has noted that the project is helping to “build identity and community…by articulating the common underpinnings of our complex and diverse organization and programmatic efforts.”  UIC’s director, in his annual progress report, says that “the most important campus outcome of the UUPP initiative at UIC in 1999-2000 has been the opportunity for the core committee to capture a working knowledge and strong sense of the vibrant UIC community.”

In addition to contributing to a more cohesive sense of the institution’s mission and of how various activities and initiatives advance the mission, portfolio development fosters understanding of the need to generate and document evidence of effectiveness in carrying out the mission.  Comments to this effect cropped up repeatedly during site visits.  UIC’s director comments that the campus’s project team has come to “an acknowledgement…that perhaps every program and initiative has an assessment piece, if we look for it.”  As IUPUI’s director puts it, “Another campus-level influence of the portfolio in terms of quality assurance is the realization that description of programs and learning is not sufficient.  All areas of the portfolio require a context of evaluation, reflection, and plans for improvement in order to be effective.”

These kinds of impacts represent real institutional change, although we do not yet have a fully developed model for maximizing these impacts.  Still, we do have some insights.  For example, involving as many people and groups as possible in the project is a key strategy; if the portfolio is treated as just another administrative project, or a mere framework for linking various campus websites, campus-wide benefits will be minimal.  In a draft article, IUPUI’s director argues that “involving faculty, staff, and committees in the portrayal of their work in the portfolio is essential to ensure the dynamic between merely writing a report and following through with the action to improve, and then demonstrating these processes for improvement on the portfolio.”  As she goes on to say, this realization is embryonic; two years is not enough time for the full impact of this work to become manifest or widely understood.  Certain conditions may need to be in place for a campus to be “ready” to benefit fully from portfolio work—for example, a developing culture of collaboration or initiatives aimed at promoting trust between faculty members and administration.

            Institutionalizing the portfolios will require even broader campus involvement than has been the case to date.  During the first year of the project, those campuses that established special committees charged with portfolio development made greater progress than campuses that hoped already-existing committees would take on this work.  Campuses that did not convene such committees during Year 1 did so during this past year, and have made substantial progress.  But for the project to have wide impact on a campus, involvement must expand beyond the core working committee.  People and groups engaged in relevant work on assessment, curriculum development, faculty development and strategic planning campus-wide also need to become engaged.

Indeed, for portfolio development to continue past the initial stages and to become a fully institutionalized process for quality assurance, work on the portfolio ultimately needs to be integrated with the ongoing work of committees and units across the campus.  This insight emerged clearly from the site visits this year.  Next year, we hope to learn more about effective approaches to making the portfolio a permanent, widely accepted campus system for improving and demonstrating educational and institutional effectiveness.

            With all of the above in mind, we have developed the following list of the benefits of an electronic institutional portfolio:

• Brings greater visibility, commitment and coherence to the institution’s mission and identity for multiple groups of internal and external stakeholders.

• Is prepared for institution’s “own” purposes, with focus on core mission areas and processes.

• Addresses important stakeholder concerns.

• Is more interactive and visual, and presents a greater variety of evidence, than paper reports.

• Serves as a catalyst for making assessment and improvement efforts more continuous, complete and cohesive.

• Presents a coherent view, with supporting evidence, of assessment and improvement efforts.

• Increases the visibility and value of key activities, including teaching and learning.

• Promotes faculty development in ways compatible with the institution’s mission.

Stages of Portfolio Development

Just as we have started to gain some insights into the organizational benefits of institutional portfolios, we also have a sense of the foundations that need to be in place to begin a portfolio and of the predictable stages of portfolio development.  Here is a summary of what we have learned about the stages of portfolio development:

Foundations for Beginning Portfolio Development

     Conceptualizing an institutional portfolio is a daunting task.  Having certain foundations in place—such as a strategic plan with clear priorities and some evidence of effectiveness—makes it easier to develop an initial organizational framework for the portfolio and to begin generating content.  Experience suggests that some, if not all, of the following foundations should be established before beginning to build an electronic institutional portfolio; with some of these in place, the portfolio may be a catalyst for developing the others.

•         A faculty-approved set of campus-wide core learning outcomes for the baccalaureate.
  

•         Some processes and structures for assessing student progress toward the core learning outcomes.
  

•        A strategic plan, or strategic planning process, and procedures for assessing progress on planning priorities.
  

•         A well-organized institutional research function that supports campus-wide planning and assessment activities.
  

•         A culture that supports collaboration across units and between faculty and administration.
  

•         A well-articulated rationale for developing an electronic institutional portfolio, including some sense of how the institution will use the portfolio and of the main audiences for the portfolio.
  

Issues in the First Phase of Portfolio Development

At the outset, a campus beginning an institutional portfolio needs to consider and make decisions about several basic issues:

•        Who should be involved in designing and developing the portfolio?
  

•         What should the major themes and focuses be?  (What is the “storyline”?) What do we know and what do we need to know to develop these themes and provide evidence of improvement and effectiveness in these areas?
  

•         How should we organize the portfolio within the web environment to communicate these themes to our audiences?
  

•         How will the portfolio be related to the university’s current web presence?

Issues in the Second Phase of Portfolio Development

•       How can we stay focused on our major themes?  What do we need to leave out?  What additional evidence do we need to include?
  

•          How can we get our intended audiences involved, not just in reacting to the portfolio, but in helping us develop content that responds to what they want to know in ways that engage their interest?
  

•         Are we using language that communicates with the audiences we want to reach?
  

•         Is the portfolio easy to navigate?  Can users readily find information they are seeking?
  

•         How can we get faculty and administrators across campus more involved?  How can the portfolio be institutionalized as a permanent quality assurance process?

The Role of Institutional Research

The institutional research operations of the six UUPP campuses brought to the project a variety of approaches, reflecting differences in institutional culture, resources and staff skills.  Facing together the new opportunities and challenges presented by the UUPP, the IR directors have shared and borrowed freely from each other.  Their collaboration has led to innovative approaches to providing information support for institutional planning, evaluation and improvement.  The portfolios provide abundant examples of these innovations, such as the Decision Support section of the GSU portfolio, UMB’s use of the NSSE results to document learning outcomes, and the CSUS key performance indicators.

The experience of campuses in the UUPP suggests that Institutional Research offices have the potential to play a broader, more central role than they traditionally play in campus efforts to enhance student learning and improve institutional effectiveness.  Institutional research offices on all project campuses have been integrally involved and have provided important leadership in developing their campuses’ institutional portfolios.  UUPP campuses report that, as a result of the project, their universities’ institutional research offices are more integrally involved in ongoing campus planning and assessment processes.

This expanded role is similar to the one that Lee Shulman and Pat Hutchings propose for IR offices in their September/October 1999 article in Change magazine, “The Scholarship of Teaching:  New Elaborations, New Developments.”  Shulman and Hutchings suggest that IR might be reconceived to play a central role in institution-wide inquiry on teaching and learning—in an institution-wide practice of the “scholarship of teaching and learning.”   We see the beginnings of this practice on several UUPP campuses; indeed, at its best, work on an institutional portfolio may enact the scholarship of teaching and learning at the level of the entire institution.

The Value of Collaboration

Finally, we should note that the collaboration among our six campuses, and with the American Association for Higher Education, has greatly enhanced the work of the project and of each campus.  An examination of the six portfolios makes this cross-fertilization readily apparent.  For example,

• Some portfolios have similar structures to others—e.g., PSU and UMB.

• GSU is adapting the department portfolio prototype developed by CSUS.

• The idea of making the portfolio interactive, first proposed by PSU, has been adopted by most of the others.

• The notion of the portfolio as a story or narrative, first proposed by IUPUI, has been adopted by UIC and UMB.

Special meetings of the campus project directors and the IR directors, not envisioned in the original proposal, played a valuable role in encouraging this cross-fertilization.  Overall, the collaboration has enriched our thinking about issues of assessment, the definition of “urban-ness” and effectiveness within the urban mission, and about all of the issues discussed above under “Stages of Portfolio Development.”  The individual campuses could not have made the progress they have if they were working alone; we certainly could not have developed a coherent set of ideas about institutional portfolios, and their development and uses, had we worked as individual institutions.

Plans for Next Year

During the next year, we will work to consolidate progress on key project deliverables, to institutionalize the portfolios on project campuses, to determine the implications of the project for accreditation and similar forms of institutional review, and to develop several dissemination vehicles intended to help other institutions interested in developing portfolios.  Important activities will include:

•         Continuing to develop the individual institutional portfolios, with emphasis on documenting a major student learning outcome and outcomes of civic engagement.
  

•         Continuing to expand internal and external stakeholders’ involvement in the project. A key initiative will be the October meeting of UUPP institution officers and representatives of major stakeholder groups.
  

•         Working toward institutionalizing the portfolios and related processes for internal improvement and external accountability.
  

•         Continuing to work toward a description of “urban public university” and to develop related indicators and measures of effectiveness.
  

•         Analyzing this year’s site visit reports and making needed modifications in site visit procedures for Year 3 visits.
  

•         Conducting a final round of experimental site visits, and analyzing the implications of the portfolios and the visits for accreditation and other institutional reviews.
  

•         Completing the Functional Needs Assessment.
  

•         Continuing to collaborate with related Pew initiatives on student learning, assessment, accountability and accreditation.
  

•         Continuing and expanding dissemination efforts, and developing dissemination vehicles to help other institutions undertake portfolio initiatives, including:

—  Continuing to present widely at national and regional conferences, including the annual conferences of the six regional accrediting associations.

—  Continuing monthly reports in AAHE Bulletin, and developing an article on the project for the Bulletin.

—  Completing the institutional portfolio section of the AAHE monograph on electronic portfolios in higher education.

—  Completing the UUPP “meta-site,” a web site that will:  provide information on the UUPP; identify and, where possible, link to relevant resources; bring together common elements of the individual UUPP portfolios; and summarize project-wide findings on the role and effectiveness of urban public universities, the process of developing institutional portfolios, and the uses of institutional portfolios.  The meta-site will replace the current UUPP web site.

—  Creating an on-line seminar on developing institutional portfolios.