Barbara Cambridge, Project Director
Associate Dean of the Faculties, Indiana University Purdue University Indianapolis
Director, Assessment Forum, American Association for Higher Education

Margaret Miller, Senior Advisor
President, American Association for Higher Education

William Plater, Senior Advisor
Executive Vice Chancellor and Dean of the Faculties, Indiana University Purdue University Indianapolis

Special Issues for Urban Public Comprehensive Universities

• Type of student body

• Cost

• Definition of institutional type

Project Goals

• Goal 1: Defining generic mission and expected outcomes
• Goal 2: Defining specific mission and expected outcomes
• Goal 3: Matching mission to public purposes
• Goal 4: Developing evidence of expected learning outcomes
• Goal 5: Creating an institutional portfolio to demonstrate the achievement of expected outcomes
  • Contrast to current kinds of documentation and review
  • Principles of institutional portfolio construction
  • Hypertextual institutional portfolios
• Goal 6: Piloting an innovative type of institution review based on the institutional portfolio and on-site audit.

Review Process

• National Advisory Board
• Institutional Review Board
• On-site audits

Project Phases

• Prior to Phase I
• Phase I: June 1998-August 1999
• Phase II: September 1999-August 2000
• Phase III: September 2000-June 2001

Project Staff

• Senior advisors
• Project director
• Associate project director
• Project assistant
• Technology development associate

Evaluation

Project Deliverables

References

Appendix

• Urban 13 Institutions

PUBLIC COMMUNICATION THROUGH INSTITUTIONAL PORTFOLIOS:
QUALITY ASSURANCE AT URBAN PUBLIC COMPREHENSIVE UNIVERSITIES

Introduction: The Need for Public Communication and Evidence

The need for public communication about the mission and outcomes of undergraduate education has never been higher. Numerous stakeholders in higher education are demanding specific information about the nature and meaning of baccalaureate degrees. These stakeholders want explicit evidence about the return on their investment in higher education.

Calls for explicitness come from both outside and inside colleges and universities. External constituents are calling for evidence that they can understand and use to make difficult decisions. Legislators, representing the taxpayers who help fund public institutions, must have rational grounds for fiscal decision-making. Competition for diminishing state funds increases the need for persuasive, evidence-based arguments about higher education’s contributions to the state and its citizens. Students and their families have two main questions: "What are the essential differences among the various kinds of institutions?" and "Amidst the plethora of choices, which college or university offers the best education and investment for me?" Needing highly educated workers, business leaders are concerned about putting more and more of their own resources into training of college graduates whom they find unready to do increasingly complex jobs. They need to know what the graduates they hire know and are prepared to do and how colleges know that to be true. Accreditors, who in the current peer review system are charged with certifying programs and institutions, are credible only if they can demonstrate progress towards explicitly expressed institutional goals.

Internally, faculty need to understand their teaching, research, and service activities in relation to institutional mission, a correlation that is new to faculty whose culture has emphasized disciplinary rather than institutional identity. With tight fiscal conditions, institutional administrative officers need clear goals and standards in order to construct and prioritize programs and activities. Clarity of mission is essential as institutions find their niches in the higher education market.

Recent history manifests an acknowledged need for evidence and also the difficulties in securing and using it. For example, a demand for comprehensive evidence about institutional effectiveness, a demand that campus-based assessment was never able to meet, is reflected in the growing use of state performance indicators. On a continuum from South Carolina, which decided to tie all funding to performance indicators, to Indiana, which is currently developing its first comprehensive set of indicators, many states require institutions to produce data that both describes their effectiveness on important aspects of institutional effectiveness and allows for comparison across institutions. Inadequate mission differentiation and inappropriate use of data, however, have caused dissension among some states and institutions about the uses of performance indicators. The kind, quantity, and quality of the information are all contested.

Another example of the need for information involves students and families who, needing evidence, have increasingly turned to college guides as resources for information in decision making. Both magazines and books that yearly rate colleges and universities are a rapidly growing business. Categories and criteria for ratings, however, have been questioned by institutions, a focus on input and reputational measures, and inadequacy of the institutional categories to capture distinctions.

Finally, some specialized and regional accreditation bodies, which gather evidence to certify programs and institutions, have recently altered their practices to acknowledge institutional diversity and to allow for alternative self-studies. Yet, institutions continue to find accreditation a time-consuming, perfunctory process that does not contribute to improving the quality of the institution through the generation of useful information revealing what they are doing well and what they need to change. Accreditors’ requirements for change may even contradict the self-understood mission of the institution: sometimes institutions are faulted for not doing something that they did not intend to do. Moreover, the results of accreditation are private and do not meet the public’s need for information.

Urban public comprehensive institutions are a particularly appropriate kind of institution to address the issue of quality assurance and communication. First, they enroll a large fraction of the total undergraduate student population in this country. Of the more than 12 million undergraduate students in all institutions in fall 1995, more than 1.3 million were at master’s-granting public urban institutions. Hence a project that develops a prototype by which institutions in this category can explain their mission and document their achievement of it has the potential to benefit large numbers of students and other clients of those institutions, such as employers.

Second, urban public comprehensive institutions have particular challenges in communicating with their publics. Although there are many aspects of their identity that create those challenges, three will illustrate the point: type of student body, costs, and definition of institutional type.

Urban publics serve the widest range of students of any category of institutions. From underprepared first-year undergraduates to doctoral candidates, their students are likely to be what are called nontraditional or new majority students, who often have different patterns of behavior and different purposes in attending the university from traditional students. Urban public comprehensives can, therefore, be misrepresented by the application of traditional measures of performance when assessing their success in serving those students.

For example, when undergraduate graduation rates, especially four-year completion rates, are used as a measure of performance, urban public comprehensives do not look effective. First, because most students are older, have families, and work part- or full-time, they often complete degrees in anywhere from six to ten years or more. They may get degrees but do so though non-continuous attendance at multiple institutions, which creates a path and timeframe not recognized by the usual time-to-degree and graduation and progression measures.

In addition, many students come with goals other than degree completion: for example, they often need to retool for job advancement or change, a goal that can be achieved with several courses that do not comprise a degree or a even a certificate. Life-long learning, which implies continuous learning over time, includes smaller numbers of courses at various times in a person’s life, not a set of courses leading to a degree. Because they are situated in urban settings where many workers and citizens live, urban publics have a particular responsibility to offer life-long learning opportunities.

Another significant area of concern for urban public comprehensives is the cost of a college education. Both actual costs and the public perception of them are at issue. The public at large and students specifically complain about the high and rising costs of a college education. Reports of the rising percentage of family income required to meet tuition costs angers both taxpayers, who assume that their tax dollars are sufficient for schools, and students, who worry about accruing significant debt to go to college.

Yet, differentiation among types of schools is rarely a part of the conversation. According to figures recently published by the Association of Governing Boards (AGB), nearly 72% of those enrolled in public and private universities in 1996-97 paid less than $6,000 per year in tuition costs. Of the public universities, urban public comprehensives present an especially low-cost option. For example, campuses in the Urban 13 (see list in the appendix) have lower tuition than the original or land-grant campuses of their multicampus universities. And they keep costs down despite a limited resource base, compared to longer-established and more traditional types of institutions. Newer urban universities have had less time to build private support, including endowments, and many experienced exponential growth just when public funding had begun to erode and funding formulae based on enrollments were replaced by restricted incentive funding in many states.

Meanwhile, the pressure on limited institutional funds is intense. Funding formulae tied to FTE student counts ignore the fact that funding for services is needed on a head-count basis, since part-time students often use as many services as full-time students do. In fact, some part-time students require more help than more traditional students because of their inexperience and need to balance personal and academic demands. And the students served by urban comprehensive institutions may require significant financial support. Per capita financial aid at the Urban 13 institutions has increased by 426% in real dollars in the past two decades.

The costs of education for the students at these institutions may be lowered by their urban location: a large percentage are commuters who do not incur housing costs, and these schools are located near the best job markets, which permits students to earn part of the costs of their education. But the lower costs to enroll at urban universities contrast with the higher costs for goods and services incurred by the campuses. Higher costs of living, land, supplies, and insurance are reflected in the 1996 Statistical Abstract that reveals a 9.4% higher cost of living in Urban 13 cities than the national average for all cities. In addition, urban campuses provide library access to the general public, have extra need for security and parking, and serve a large population of students with disabilities - who come to urban comprehensives because they can live at home, locate employment that accommodates disabilities, and find support services. In some state capitals, state employees are offered as part of their job compensation, are waived tuition at the urban public comprehensive institution, with no recompense to the university.

The 1997 report of the Council for Aid to Education , "Breaking the Social Contract: The Fiscal Crisis in Higher Education," identifies what it calls "a catastrophic shortfall in funding" for higher education. One of its recommendations to address the shortfall "greater mission differentiation" in the restructuring of colleges and universities. Such restructuring will require accurate and full information about various institutions for valid decision-making, both internally and externally. Both the effective transmittal of current information and the generation of new information are essential in making good decisions in the present and future. This is especially true for urban public comprehensives, whose missions and circumstances are currently so inadequately represented.

A third significant difficulty for urban public comprehensives is the lack of a clear definition and public recognition of its institutional type. Performance measures that make no distinctions among institutional types are the extreme version of the problem, but even the Carnegie categories which attempt differentiation do not recognize the distinct features of urban public comprehensive universities. When performance indicators are used for funding and the Carnegie categories for comparisons by ratings systems used by consumers of higher education, urban public comprehensives suffer.

Generation of the data that would support the differentiated categories necessary to define the urban public comprehensive requires institutional research (IR) capacity that urban institutions typically lack. Despite a start on collecting data across the Urban 13 in response to requests form the National Commission on Costs of Higher Education, staffing limitations slow the effort to respond to such requests for information. For example, among eighteen Urban 13 institutions reporting, ten institutions had two or fewer full- or part-time IR personnel. Although three institutions had four persons, at least two of those persons worked full-time generating data for their university system. This project would add to that capacity and pool the research capabilities of the institutions.

Provosts identified key data elements that are particularly important to urban public comprehensives. The urban setting provides many opportunities for students to use the city as a laboratory and a learning opportunity: these opportunities should be reflected in data about internship and service-learning placements, for instance. Other information pertinent to urban public comprehensives includes crime statistics, support services for underprepared students, facilities costs because of extended hours for evening and weekend classes (one provost compared his campus to an airport that needs to clean its facilities most hours of the day), real costs associated with part-time students, and the cost of smaller classes required because of lack of student preparedness and space acessibility. It is clear that the need for data collection and analysis in these and other areas is urgent for benchmarking and for communicating with external audiences.

This project brings together six urban public comprehensive universities --California State University, Sacramento; Indiana University Purdue University Indianapolis; Georgia State University; Portland State University; the University of Illinois, Chicago; and the University of Massachusetts Boston – to enhance their capacity to communicate their mission and the attainment of that mission publicly, particularly that aspect of the mission that is focused on undergraduate learning. They will do this by:

When the public thinks of "the college" or "the university", there are clearly a couple of pictures they have in mind: for "the college", a Liberal Arts 1 institution with a residential, full-time student population of traditional age and ivy-covered walls; for "the university", a Research 1 public flagship campus with a national mission and national or international scholarly reputation. In that mental context, it is very important that urban public comprehensive universities define their distinctive nature and character. Institutions have individually begun to describe themselves, yet disparate definitions do not enable the kind of change that is necessary in everything from informing public perceptions to generating a new Carnegie classification to differentiating performance indicators. For example, the provost of the University of Memphis calls urban universities the land-grant universities of the 21^st century. Does the claim of extending a land-grant tradition differentiate or blur the category of the urban public comprehensive university? Georgia State University describes itself as an "urban research university." Is Georgia State committed to doing research on the urban setting, or are the words urban and research separate descriptors? The consequence for mission definition is enormous.

Urban public comprehensives, like other colleges and universities, must instantiate their mission through their actions and demonstrate their success in achieving it through their results. Although some urban publics offer significant graduate programming, often linked to the undergraduate mission through interdisciplinary alliances and undergraduate research, a common goal of every urban public comprehensive is to provide a high-quality undergraduate education., how it will ensure it, and how it will know it when it sees it, it provides for itself an essential basis for decision making.

Although quality has been defined in many ways, the 1995 Education Commission of the States report "Making Quality Count in Undergraduate Education" is helpful in identifying two common themes in most definitions, one addressing abilities and attributes of college graduates and one describing attributes of the learning environment. Its list of desirable student outcomes includes "higher order, applied problem-solving abilities; enthusiasm for continuous learning; interpersonal skills, including communication and collaboration; a strong sense of responsibility for personal and community action; ability to bridge cultural and linguistic barriers; and well-developed sense of ‘professionalism’" (5-6). The same report lists, as attributes of the learning environment, such "quality-oriented institutional characteristics" as student-centeredness, commitment to specific ‘good practices’ instruction, ‘quality management’ practices, and efficiency and integrity of operation" (7-8). Although these descriptors are only examples, the practice of stipulating outcomes and characteristics inherent in mission emerges as essential for assessing and communicating quality.

Urban public comprehensive universities can benefit from comparing among themselves the specific outcomes of undergraduate education that they aim to produce and the characteristics and practices that foster those outcomes. In doing so, the institutions will be working toward a common definition of undergraduate learning that is distinctive to their type of institution. The fact that urban universities serve non-traditional students, for instance, could suggest a focus on the development of competencies for specific workplace needs; a largely part-time student population might drive the development of out-of-classroom learning experiences; and the needs of place- or time-bound working students might suggest that the enabling environment should include forms of asynchronous instruction. Part of the work of the grant will be identifying and providing evidence about the common features which distinguish urban public comprehensive universities and influence the results that they aim for and produce.

In addition, however, the project recongnizes the individuality of campuses and subsets of campuses within the project. Depending on history of the institution, position in a system, context of community, and leadership within the institution, each campus will also want to represent itself in different ways. For example, the University of Illinois at Chicago exists in a highly competitive market for higher education institutions, so that differentiation not only by institution type but by services in the context of Chicago is important to the campus. A subset of two campuses in the project, University of Illinois at Chicago and IUPUI, both have medical schools, whose presence on their campuses significantly influences their mission.

Differentiation, therefore, will occur on three levels: the mission will contain categories common across colleges and universities, features within the categories distinctive to urban public comprehensive universities, and features particular to a specific urban public comprehensive university. This three-tiered approach applied to curriculum, pedagogy, and faculty work is illustrative.

In the 1997 Handbook of the Undergraduate Curriculum, Peter Ewell notes that performance indicators focus not only on outcomes but on different functions that contribute to the outcomes. One of those functions, for example, is a coherent and integrated curriculum, a goal for all colleges and universities. Yet, coherence and integration may well be defined quite differently by a four-year liberal arts school with residential students and an urban comprehensive university with commuting, part-time students. For example, the presence of a significant number of adult students with short-term educational objectives may mandate development of one-and two-year certificates. And individual campuses may exhibit this element of the urban public comprehensive mission differently. For instance, the University of Illinois at Chicago, in a city of museums, may offer a museam-studies certificate whose classes are offered on weekends and located at a different museum each Saturday.

Curriculum is, of course, only one way universities generate learning—effective teaching strategies are also important. Because of increased knowledge about ways that people learn, most colleges and universities are now stressing active learning. Within the rubric of active learning, urban public comprehensives might choose to focus on active-learning strategies that also meet the social needs of commuting students, such as learning communities. Or the urban setting might lead to the development of active-learning strategies that make use of the rich urban environment, such as internships and service learning. Were participating institutions to decide that service learning were a common pedagogy particularly appropriate to all urban public comprehensive universities, indicators such as number of students participating, differential in grades of students involved and assessed contributions of service learning to the understanding of the content of the course could be developed. Each institution, in addition, might have its own special opportunities to foreground: for example, Portland State has a complex relationship with numerous community partners for student placements in applied learning situations.

Just as the traditional Carnegie categories do not fit urban institutions, current categories of faculty work may need to be changed to describe accurately what faculty at urban public comprehensives do. Traditional notions of the balance among and nature of research, teaching, and service are becoming increasingly problematic for all institutions, but for urban research institutions in particular. For instance, a common feature of urban comprehensives may be that research by faculty is or is encouraged to be applied. Applied research blurs the lines in the traditional attribution of faculty work time between research and services. The Boyer categories of scholarship – integration, application, discovery, and service – may be the best way to evaluate faculty work at urban public comprehensive universities. More traditional institutions have been slow to adopt these categories; it may be that a mark of urban comprehensives will be their embrace of the categories as significant markers of their institutional type. In addition, however, each institution may foreground certain kinds of applied research. At IUPUI, for example, clinical research that involves both the medical school and the school of science is a strength of the university.

These examples of curriculum, pedagogy, and faculty work only scratch the surface of categories that participating institutions will want to take up to develop core and institution-specific indicators and measures. Part of the exciting work of the project will be the discovery of which categories are most appropriate for accurately describing and evaluating urban public comprehensives.

One limitation of traditional forms of institutional self-examination that this project is designed to overcome is their characteristically self-referential nature. It is time that the academy stopped talking only to itself and invited others in on the conversation. Higher education’s failure to do so has led to attempts by those outside the institution to develop their own ways to measure institutional effectiveness. The state-level performance measures that have been one result of this public frustration have been largely incoherent, have contained public-polity assumputions and entailments that have not been carefully thought out, and have not proven useful to the institutions whose results are being described.

The incoherence of state performance indicators is underscored in a recent study of them by Joseph Burke of the Public Higher Education Program at the Rockefeller Institute of Government. Burke found a startling scarcity of common choices in the type of performance indicators across states: in fact, only five of 58 indicators were shared by four of the eight states in the study. Moreover, they contained evidence of the lack of communication between public representatives and the institutions they were portraying. Overall, the external concerns of states and society influenced 68% of the indicators; the internal concerns of the academic community 22%; and combined concerns 10%.

In the study Burke identifies three models of excellence: resource/reputation (primarily a traditional faculty-oriented model), strategic investment/cost benefit (primarily state-oriented model), and client-centered (primarily a student and other consumer-oriented model). A clear trend is developing for the merged model of strategic investment and client-centered. Within that merged model, Burke notes a declining interest in inputs, a growing attraction to outputs and outcomes, and "a surprising acceptance of process measures. The process measures indicate in part a growing concern with improving institutional processes that flows from the quality and good practices movements."

Two conclusions in the report bear directly on this project: (1) "Identification of preferred values and desired models for public higher education should precede and shape the choices of performance indicators," and (2) "Performance indicators should avoid the artificial and adversarial distinction between external accountability and internal improvement by choosing indicators that combine the strategic investment and the client-centered models." This project will identify the "preferred values and desired models" for urban public comprehensive universities by the cultivation of agreement among participating universities. This should result in a core set of common performance indicators specific to this institutional type that will both be useful to the institutions and further the work on performance indicators within and across states, if and when states recognize the necessity of differentiating indicators among institutional types. The interest of states in the project-developed indicators should be increased by the fact that they were produced by a working alliance of institutional representatives and external constituents and represent a balance between the needs of society, the institution, and the student.

Just what kinds of evidence will the project aim to produce to demonstrate the universities’ achievement of their learning mission? The four categories of evidence include direct measures of learning, other measures of attainment of value to various constituents, good practices, and an enabling environment.

1. Direct measures of learning. These measures shows what learning the institution is certifying with the degree. Some examples of direct measures are test results, evaluations of authentic performances, and student portfolios.
2. Other measures of attainment of value to various constituents. Outputs, such as retention and time to degree in the context of the characteristics of entering students, indicate are of interest to state-level stakeholders. Outcomes, such as success in further study, graduates’ satisfaction with preparation for work and citizenship, graduates’ behavior as workers and members of communities, and employer satisfaction with graduates, are all indicators of the institution’s level of success that are of particular interest to students and employers.
3. Good practices. These measures are examples of practices that research suggests contribute to student learning objectives. Some examples are active-learning opportunities such as internships, undergraduate research, service-learning, and collaborative learning.
3. Enabling environment. Features of an environment that enables student learning might include ongoing faculty development opportunities; faculty productivity in the scholarly areas of discovery, integration, application, and teaching; administrative practices that support the learning mission; availability of current technology; cross-functional instructional teams; and a physical plant conductive to student learning inside and outside the classroom. Some important parts of this environment specific to an urban public comprehensive university might be childcare facilities and security measures.

The task of generating information useful both internally and externally is a significant one. The project, however, need not start from scratch. "Student Outcomes Information for Policy-Making," a report of the National Postsecondary Education Cooperative (NPEC), for example, identifies principles by which information can be screened. First-screen criteria, slightly adapted here to the goals of this project, include relevance (how closely an outcome is related to the institution’s mission and an important policy issue), utility (potential usefulness for institution and agency), and applicability (extent to which the information meets needs of multiple stakeholders/user groups). Second-screen criteria include interpretability (likelihood of understanding by multiple users), credibility (level of trust of different groups in information on an outcome), fairness (balance of perspective among groups), scope (size and breadth of sample), availability (existence, accessibility, and feasibility), measurability (operational definition, reliability, and validity), cost (appropriateness of financial expeditures). Although these criteria were aimed primarily at state-level policy issues, they are a set of principles institutions in the project will consider as they identify the evidence that will serve internal and external needs.

Participating institutions will agree on at least some common outcomes for undergraduate education. For example, one desired outcome currently identified by many internal and external stakeholders is "ability to work in collaborative groups." First, the portfolio (see goal 5, below) would include a definition of the desired outcome, such as "Ability to work in collaborative groups involves experiences and expertise in leadership and cooperation in accomplishing a mutual task." Among the "good practices" listed might then be general education courses and co-curricular activities in which students practice and are evaluated on their collaborative abilities. Regardless of the pedagogies and learning experiences of the students that are different on each campus, institutions will strive to identify common assessment measures to evaluate outcomes. The more important point for external use is the assessment data that shows student progress toward the goal, although how they reached it or why they did not will be essential for internal decision making.

Like an individual student portfolio or a teaching portfolio, the institutional portfolio serves a double purpose as a chief learning tool and as a form of communication. Portfolios are particularly appropriate for this double purpose of learning and documenting learning because they are part of a process of self discovery, examination, and reform, as well as a record of that process and its outcomes.

The process of doing a portfolio reveals successes and weaknesses which can be addressed as the portfolio is being constructed. The second function of the portfolio, nonetheless, is more central to the goals of this project. The institutional portfolio will demonstrate institutional effectiveness in ways that are warranted by both internal and external audiences. Developing institutional portfolios that are useful in multiple contexts is a challenging goal.

The institutional portfolio will contain the kinds of evidence described above under Goal 4. The core indicators of mission acheivement and the general organization of the portfolio will be collectively determined by the participating universities and reviewed by the Institutional Review Board (see page 26) of this proposal for a description of the board). Some elements of the portfolio will be uniform; others will vary according to internal or external characteristics particular to the individual institutions.

Two kinds of documentation currently produced by institutions are self-studies for program review and self studies for accreditation. The self-study for program review usually focuses on one unit within the institution. The review of that self-study may have the purpose of evaluating the unit for funding or continuation and/or the purpose of determining areas for development or change. When the former purpose prevails, the self-study understandably reveals only the best of the unit because it is justifying its existence: the main purpose is not to assess for improvement. When the latter purpose prevails, the self-study may indeed be useful to the unit studying itself, but the criteria and language of the report need only be self-referential. Both purposes are internal to the institution.

The second kind of documentation, the self-study for accreditation purposes, presents evidence to meet minimal standards established by an accrediting association to which the institution belongs. Reviewers check compliance to standards set by the association. Institutions prepare accreditation self-studies every ten years for regional accreditation and at varying intervals for specialized accreditation. This self-study and the review process that ensues involves reviewers who are outside the institution but still inside the world of higher education. Thus, this process too is, in the final analysis, self-referential, and the results are private. Often the one effect the outside reviewers in the specialized accreditation process have is negative: to the consternation of presidents and provosts, such reviews can force reallocation of resources within institutions towards the program under consideration, without respect to the larger institutional picture.

The institutional portfolio envisioned for this project serves multiple purposes for multiple stakeholders. Instead of disparate documents, sometimes redundant and often undervalued internally if they are solely for external purposes and incomprehensible to outsiders when they are produced for internal purposes, the institutional portfolio will enable institutions to self assess, self adjust, and represent themselves in a way that meets both internal and external needs. The portfolio can be constantly revised and kept current as a resource for reform and for providing information to interested stakeholders about both what students are learning and how the institution is supporting that learning. And the external review of the portfolio and the audit represented by the site visit will also serve the double purpose of public communication and improvement. In contrast to specialized accreditation, in this case an external evaluation can be used to leverage internal change for the good of the entire institution rather than of particular programs within it.

For the portfolio and review to serve the internal purpose of improvement and the external purpose of explanation, writers and readers of the portfolio will need to understand both internal and external goals and criteria. For example, faculty members will need to be aware of standards of accreditors and the expectations of states, while reviewers from business and government will need to understand elements of the learning environment necessary to support student learning. The people involved in this project represent those constituent groups that must be involved in the institutional portfolio process for it to be successful.

The National Center for Higher Education Management Systems has developed a set of principles for construction and use of institutional portfolios that can serve as a starting point for the project. Although the larger set of principles were developed with accreditation in mind, some principles can be adapted for this project.

1. Institutions should develop their own models for organizing and presenting their portfolios. In the case of this project, institutions will collectively develop the model for organizing and presenting the portfolio, enabling the kind of analysis and evaluation that needs to be done across campuses, and within that template each institution will have room to document its unique aims and results.
2. Portfolios should contain guidance about what is being claimed and in what way. The institutional portfolios will provide interpretations of the data that they contain.
3. Data and exhibits may be qualitative or quantitative. The institutional portfolios will contain both types of documentation. Participating institutions will decide on common types of documentation, both quantitative and qualitative, as well as the kinds of institution-specific evidence they want to include.
4. The totality of the exhibits is the best indicator of institutional effectiveness. In these portfolios, no one piece of documentation about any of the essential elements will need to suffice.
5. The contents of the portfolio are subject to verification, so backup documentation and activities may be part of the external review. This project includes as a major component an innovative kind of external review. Participating institutions will agree upon a set of principles to guide this review.

A kind of formatting that is suggestive for the construction of institutional portfolio sections comes from the state of Virginia. The State Council of Higher Education for Virginia is in the process of publishing as series of documents designed to address the questions of higher education’s stakeholders by providing them with varying kinds of evidence. For example, a 1996 report on "What do students learn?" presents two-page narratives for each institution reporting how assessment programs have helped in efforts to improve the effectiveness and efficiency of teaching and learning on each campus. A 1997 report on "What do students experience?" presents graphical data about types of students enrolled and their academic experiences, including what they study, who teaches them, class sizes, and the number of courses that require integration and application of learning. Charts and bulleted lists present selected information about each institution. These formats were chosen to accomplish the purpose of this series of publications: "to give constituencies of higher education—students, parents, legislators, the business community, faculty and staff, and the public—the information they need to make good decisions about Virginia’s colleges and universities. Together (these publications) provide a comprehensive and meaningful overview of public higher education in Virginia.

One answer to the need for a reasonable amount of material, for variety of presentation formats (including multimedia), for both core and institution-specific measures and indicators, and for evidence on which particular audiences can focus is the hypertextual institutional portfolio. This on-line format would enable institutions to create what Mark Shadle of Eastern Oregon State College calls a "computerized mothership portfolio" as the basis for its multiple uses of materials. This would mean that reviewers, instead of progressing in a linear fashion through a prescribed set of materials, could click until they saw enough evidence to be satisfied about a particular area being evaluated. While problems of availability of technology, the computer competence of reviewers, costs of time and money, and other variables all need to be considered, the institutional portfolio as hypertext fits other developments in using the Web for better communication.

For example, the proposed Carnegie Foundation Teaching Academy includes the development of a Media and Dissemination Lab to explore new forms of documentation and dissemination involving the scholarship of teaching. Randy Bass, a consultant for the AAHE Peer Review of Teaching project, is already directing a national project in which course portfolios are being put up on the Web. Pat Hutchings hopes that the CRAT Lab will be "just what’s needed" to make on-line portfolios "the norm rather than the exception five years from now."

Collaboration with Carnegie Lab would enable the six urban public comprehensive universities involved in this project to either create their institutional portfolios hypertextually from the beginning or to envision moving to that format in the future. The two projects could also be linked: institutions that wanted to give reviewers a sense of how learning objectives are built into the curriculum might construct links on their institutional portfolios to course portfolios of faculty who are using new learning strategies; those portfolios in turn might link to student portfolios that graphically demonstrate the learning outcomes of the new approach.

Another resource for the project is the team at California State University Monterey Bay, which has produced a CD-ROM as part of its self-study for accreditation. Both John Ittelso, director of distance and online learning, and Linda Stamps, director of the accreditation self-study team, stand ready to consult with project participants who desire to prepare a hypertextual portfolio.

The project’s technology development associate would have the responsibility of developing a functional Website for posting of materials and progress reports from the various campuses, but he or she could also help prepare portfolios for later hypertextual formatting. For example, setting up print text styles that adjust well to screen displays and creating a visual archive are two practices that all institutions can adopt. This part of the project has enormous potential for influencing the ease of public communication.

Reaching agreement on common goals and measures will be a challenge for the participating institutions. The task becomes more valuable, however, when institutions know that the regional accrediting associations are open to using the institutional portfolios for the current accrediting process. Jean Avnet-Morse, Middle States Association; Peggy Maki, New England Association; Steve Spengahl, North Central Association; James Rogers, Southern Association; and Ralph Wolff, Western Association have all expressed interest in the project and expect that the institutional portfolios produced could substitute for part of the current accreditation alternative self-study.

Traditional accreditation in this country, however, has been a private affair whose chief aim has been institutional improvement. Numerous calls have been made for the development of a peer review process for quality assurance that both improves conditions for learning and informs publics. In a 1995 "Accountability Study," Patricia Albjerg Graham, Richard Lyman, and Martin Trow recommended "strengthening of the internal processes whereby faculty members and administrators collectively assess the quality of and make improvements in academic programs" (18). The authors suggested that accrediting bodies should audit the institution’s own internal quality-assurance processes and improvements based on the findings of those processes. "External accountability should be focused on assuring the presence and effective functioning of internal mechanisms of accountability" (18).

In 1996 four eminent scholars from the United States and England made the case even more strongly: "What is needed is a coordinated effort of self-regulation that would require all postsecondary institutions to allocate faculty time and other resources to academic quality activities as a fundamental business requirement. . . . This coordinated effort should be the primary focus of any future reforms in accreditation." The authors recommend that the Council on Higher Education Accreditation (CHEA), a national body of presidents concerned with accreditation, "encourage the development of institutionally-based teaching and learning quality assessments and experimentation with academic audits" (24).

The innovative review process in this project builds on the productive aspects of each of three basic approaches to quality assurance: accreditation, assessment, and academic audit (Dill, Massy, Williams, Cook). Accreditation by an external agency uses established criteria to judge through an institution’s self study and a review visit if the institution meets minimal standards. Assessment, either external or internal, describes the quality of specific activities or outcomes in reference to a unit’s mission or goals. Academic audit, externally driven, focuses on processes which produce quality and assessments that academics use to assure themselves that quality has been achieved.

This project builds on the strengths of all these approaches: it ensures that accreditiation can acknowledge and support more than generic minimal standards through the resources of the institutional portfolio, it provides a context in which the meaning of any set of assessment results can be determined by comparison with the results of like institutions, and it ensures that the academic audit’s outside reviewers are both informed as to institutional mission and represent a perspective that is outside of higher education. The people doing the review, the nature of the evidence they examine, and the protocols of the site visit will combine into a genuinely new form of quality assurance.

Choices about assessment measures and especially the way in which the results are interpreted and reported in the institutional portfolio will benefit from the participation of the two advisory boards, which play a central role in the project.

Program review in the past primarily involved internal readers and accreditation primarily involved external readers from within higher education. This project significantly alters the makeup of the reviewers of institutional portfolios, however, by involving constituents knowledgeable about but external to higher education on the portfolio review teams.

The project creates two groups with a significant number of external constituents. The National Advisory Board (NAB) will include distinguished persons from government, business, foundations, and higher education to advise the project about its aims, practices, and progress. The voices within this Board will represent significant players in the future of higher education from the perspective of funding and public endorsement of higher education’s achievements. Possible members of the Board include Peter Ewell, National Center for Higher Education Management Systems; James Fallows, editor, US News and World Report; Edward B. Fiske, editor, Fiske Guide to Colleges (check title); Gerald Baliles, former governor of Virginia; Joseph Dionne, Chairman and CEO, The McGraw-Hill Companies, Barry Munitz, president, the Getty Foundation, Judith Ramaley, president, the University of Vermont; Jim Mingle, executive director, State Higher Education Executive Officers; Ralph Wolff, director, Western States Association of Colleges and Universities; Patrick Callan, director, Higher Education Policy Institute; B.J. Overton, program director, Kellogg Foundation; Peter Magrath, president, NASULGC; Tony Carnevale, vice president for public leadership, ETS, and Dolores Cross, president, the General Electric Foundation; and a undergraduate student representative.

At annual meetings, this group will review the evolving set of core goals, indicators, and measures; provide project participants with current information about the emerging perspectives of the groups the Advisory Board represents; and become itself more acquainted with the issues faced by urban public comprehensive universities.

The Institutional Review Board (IRB) will be even more hands on. It too will be comprised of government and business leaders, along with university trustees, third sector representatives, members of accrediting bodies, and higher education leaders of institutions and other public comprehensives. Possible members of this board would be Peter Ewell, National Center for Higher Education Management Systems (other crossover members between the two boards may also be considered); Carol Whitcomb, president, Foundation for Independent Higher Education; Gene Tempel, executive director, Center of Philanthropy; Ken Gladish, executive director, Indianapolis Foundation; Cynthia Davenport, executive director, Association of Specialized and Professional Accreditors; Carol Cartwright, president, Kent State University; Tito Guerraro, president, Southern Colorado University; Joan Girgus, chair of psychology, Princeton University; Elaine El-Khawas, professor, UCLA; and government and business leaders nominated by project institutions. Each IRB member will become knowledgeable about one project institution, advising it on the construction of its portfolio, especially about the kind, quantity, and presentation of evidence. That person will be involved in a pre-audit of the institutional portfolio of her assigned institution during Phase II of the project. During Phase III, each IRPB member will participate in the full review of another institution. All IRB members will attend project annual meetings to contribute ideas and expertise to all participating institutions during all three phases of the project.

Dill, Massy, Williams, and Cook predict widespread benefit from projects like this one which alter the current review process for institutions: "If several solid experiments with academic audit would be pilot-tested, we believe the practice would likely spread throughout the higher education system" (24). Project participants fully expect that lessons from the project will influence both urban public comprehensive institutions and other colleges and universities striving to ensure the quality of their undergraduate degrees. This project might well serve as a model for reform in review practices for other categories of institutions.

This project anticipates reform of accreditation practices by developing comprehensive documentation of the essential elements of an undergraduate education, by including new partners in the peer review process, and by piloting a collaborative audit process. The current emphasis by regional, specialized, and professional accrediting bodies on outcomes assessment is a step in the right direction, but description, interpretation, and evaluation of the aligned complement of goals, learning environments, and assessment methods for the purpose of improvement as well as accreditation would be a giant leap. The NASULGC Ad hoc Subcommittee on "The Student Experience and A Learning Society" has called for sharing knowledge and resources about developing independent learners " that are developed by institutions with other institutions and the public at large." This coalition of partners will strengthen the credibility of the review process that accredits postsecondary institutions. In addition, it will enable more accurate and understandable information for the variety of publics which need to make decisions based on institutional effectiveness.

Each project institution will undergo an on-side audit during Phase III of the project. The audit is designed to augment and to confirm evidence presented in the institutional portfolio. During Phase II, institutions will have a pre-audit with the IRB members most familiar with their campus and with the institutional portfolio. During Phase III, institutions will have an official audit with NAB members and IRB members who have not previously worked with the campus. After the pre-audit and the audit, project staff will debrief participants and reviewers to determine the elements of a successful audit. This process will furnish information about how to prepare for and conduct an audit.

1. Selection of participating institutions:

In October 1997, Ellen Wert, Margaret (Peg) Miller, William Plater, and Barbara Cambridge met with prospective provosts from four selected Urban 13 institutions, at which time three institutions committed to participation: Georgia State University, the University of Illinois at Chicago, and Portland State University. At the same time, on the recommendation of Ralph Wolff and because of the desire for a Western States participant in the project, Barbara Cambridge contacted the provost and president of California State University at Sacramento, who indicated their commitment to the project.

2. Solicitation of interest from regional accrediting associations:

In October 1997 Peg Miller and Barbara Cambridge contacted regional accreditors to solicit their participation. All five regional accrediting bodies contacted expressed interest in the project.

3. Establishment of the role of university presidents/chancellors:

Gerald Bepko, IUPUI Chancellor and Vice President for Long-range Planning for Indiana University, will convene by videoconference the presidents or chancellors of the six participating schools to discuss the role of the chief executive officer in the success of the project. Particular attention will be given to uses of current campus advisory boards in development and local review of institutional portfolios.

4. Selection of members of the National Advisory Board and the Institutional Review Board:

Participating schools will be asked to review the proposed members for the two boards. After selection of invitees by the project staff, letters of invitation will be extended. Invitations will be sent until the boards are fully constituted.

5. Selection of campus project directors:

Each campus will select a campus project director with a three-year commitment. The project director will devote at least ½ time to the project.

Phase I includes many more all-participant meetings than either Phase II or Phase III because building consensus about the mission of the urban public comprehensive university, the core indicators, and the core measures is crucial to the success of the project.

August 6-8, 1998: The goals of this meeting of a team from each campus (provost, campus project director, institutional researcher, and other designated members) members of the two advisory boards, and the project staff will be to draft the first working definition of "urban public comprehensive institution" and to begin to develop a portfolio template.

Before this meeting, provosts and campus directors will have identified campus groups which will work on the project and prepare a status report on current student learning goals, key features of the campus learning environment, and current assessment practices. In addition, provosts will bring to the meeting the accrediting guidelines and state performance indicator requirements applicable to their campuses.

October 1998: Provosts, project directors, and institutional researchers will meet at the fall Urban 13 meeting. With campus input, a working definition will be revised. A portfolio template will be more fully developed. The group will decide on core indicators that apply to all campuses on which they will seek institutional affirmation during the ensuing months. IR personnel will meet separately to discuss data collection.

December 1998: During a videoconference hosted by IUPUI, campuses will refine the definition of "urban public comprehensive university." They will continue to develop the categories of goals, indicators, good practices, and enabling environment.

April 1999: During a meeting at another participating institution, participants will decide on a limited number of common indicators and measures. They will further refine the portfolio template.

August 1999: This annual summer meeting of the campus participants, project staff, NAB members , and IRB members will have four objectives:

• Achieving consensus about the definition of urban comprehensive university as an institutional category useful for multiple purposes
• Achieving consensus about the portfolio template to be used in the next phase of the project
• Sharing information and lessons from campus activities during Phase I
• Planning the work of Phase II

All project participants study one another’s processes for aligning goals, learning environment, and assessment measures; hallmarks and barriers identified by each institution; new practices which they might adopt; and the evolving institutional portfolio template. The IPRP will participate for the full meeting; the NAB will participate for one day of the meeting. Participants will plan any cross campus visits and IPRP visits for the coming year.

1. Each university will conduct a campus-wide discussion of core goals and institution-specific goals that support the mission of the urban public comprehensive university. This discussion should also go beyond the campus to involve important local external constituents, such as the business community and state policy-makers.
2. Each university will conduct an inventory and evaluation of existing practices that support the core goals, around which the campus will design its portfolio. The portfolio will demonstrate the alignment of the core learning curricular and pedagogical practices, and the enabling environment, with the whole governed by the mission of the urban public comprehensive university.

Within this more comprehensive audit, each institution will feature two hallmark practices as illustrations and examples of its commitment to undergraduate learning.

• Good practice hallmark. This strategy will be a central, campus-wide active learning strategy that demonstrates the campus’s commitment to student learning. Examples include a redesign of bottleneck introductory courses in a technology-based format, service-learning in all disciplines, undergraduate research engaged in by most students across disciplines, extensive student participation in learning communities, or undergraduate portfolios that accentuate and assess integrated learning.

• Enabling environment hallmark. This strategy will be a campus-wide practice that supports a comprehensively-developed learning environment for student learning. Examples might include integration of academic and student affairs to support student learning with documented results, use of cross-functional teams for problem solving, funding mechanisms that support cross-disciplinary general education curriculum development and assessment, or effective shared governance.

3. Each university will identify impediments to the alignment of the core and institution-specific learning goals with campus environment and develop a plan to remove impediments. These barriers might include misplaced emphasis in the faculty reward structure, overemphasis on majors over general education, lack of structure for campus-wide curriculum decisions, insular administrative units, or lack of consensus on assessment measures. Each university will address one comprehensive barrier during the project and develop a timeline for completing the removal of additional impediments. Participating universities may decide to focus on a common barrier during the project.
4. Each university will conduct an audit of its existing assessment and measurement practices and identify new ones that are needed to generate evidence of outcomes. Participating universities, or groups of two or three universities, may decide on some common assessment techniques. For example, the project may develop an exiting student survey to parallel the entering student survey being developed by another Pew-funded project (The University of Houston, IUPUI, Portland State, and Temple), a survey that will be available for testing by this project’s participants in Phases II and III. The entering and exiting surveys will address those issues most appropriate for an urban public comprehensive university.

    

1. Project staff members will plan and facilitate all-project meetings.
2. A Web page will be a primary site for communication among participating institutions. During Phase I, the technology development associate and the associate director will develop the initial site for information sharing. The technology development associate will begin work on developing hypertextual portfolios.
3. The project director with the IRPB member assigned to the campus will make one site visit to each campus to support campus work.

From September 1999-January 2000, each campus will

1. Meet with a representative of its regional accreditation association to discuss the evolving institutional portfolio’s use in accreditation
2. Modify and document modification of the learning environment based on what it is learning in the project
3. Continue to focus on its hallmark practices, fully implementing and documenting them
4. Arrange a spring pre-audit with its IRB assigned member and members of the local community

From February 2000-July 2000, each campus will

1. Modify and document modification of practices based on what it is learning in the project
2. Modify its portfolio as it develops information
3. Conduct a pre-audit of the portfolio with its IPRP member and local QAAB Board

1. August 2000 - The second annual summer meeting will feature analysis of the institutional portfolios. Each campus will bring six copies of its portfolio to the meeting. Prior to the meeting, the project staff will assign to small discussion groups formed for the purpose one or two cross-cutting problems that are evident, display of evidence, readability/usability, and adaptation to needs of multiple audiences. Groups will work on these problems at the meeting and prepare written reports with their recommendations about how to deal with them.

Also prior to the meeting, the staff will decide on consultants to work with the small groups and to help campuses address other problems at the summer institute. For example, if campuses wanting to achieve consensus on measures for assessment of critical thinking need help, Dave Porter from the Air Force Academy, which has done extensive work on developing instruments in that area applicable to military students, might be brought in. If representing institutional mission for the media is problematic, Rick Love of the Knight Foundation might be able to supply an educational journalist to consult with project participants.

The Urban 13 provosts have invited this project’s participants to report at each of the fall and spring meetings of the Urban 13 during the course of this project. Meetings could, therefore, be attached to both Urban 13 meetings. The needs of the project by that time will drive the content of the meeting, but certainly project participants will want to compare portfolio contents and format. Also the institutional researchers who are generating data will want to agree on their methods, interpretations, and presentations.

3. Phase II project staff activities

1. The project associate director will continue to be responsible for communication among the project participants, especially on the Web.
2. The project director will participate in each of the pre-audit meetings to collect information of benefit for all campuses.
3. The project staff will prepare for the publication of lessons from the project. Preliminary lessons from the project might appear in the AAHE Bulletin, newsletters of organizations and associations represented on the advisory boards, and campus newsletters. Plans will be laid for the monograph which will present the institutional portfolio model.
4. The project staff will consult with directors of the regional accrediting associations about ways to disseminate lessons from the project at the association’s annual meetings and board meetings which consider accreditation standards and practices.
5. The project staff will meet with SHEEO representatives about ways to incorporate lessons from the project into state indicators and performance measures.

1. Between September and January, members of the IPRP will conduct institutional portfolio and conduct the on-site audit. Each review will have an accrediting association observer.
2. Institutions will revise and refine their portfolios depending on conclusions from the review.

3. Institutions will continue to devise ways in which their portfolio can be used for various audiences and will share parts of the portfolio with those audiences for initial feedback.
4. Project staff will work with institutions to determine appropriate dissemination vehicles for the project model. Possibilities include an AAHE monograph on institutional portfolios, workshops at regional and specialized accrediting meetings, hosting the annual Urban 13 meeting, presenting sessions at other higher education meetings (such as AACU, NASPA, ACPA, ASPA, and AREA), and presenting sessions at meetings of disciplinary associations.
5. The project will culminate in a day-long conference preceding the June 2001 AAHE Assessment Conference. The NAB, the IPRP, project staff, and participating institutions will all report on outcomes of the project. Invited participants in the all-day conference will include members of audiences who will benefit from institutional portfolios, especially targeting external audiences; however, an open invitation will be extended to people in higher education, especially constituents in urban public comprehensive universities.

Margaret Miller, President of the American Association for Higher Education (AAHE), and William Plater, Executive Vice Chancellor and Dean of the Faculties at Indiana University Purdue University Indianapolis (IUPUI) will be senior advisors for the project.

AAHE is the project’s host association. It will contribute to the project by providing staff members as consultants to the project when appropriate, advising the project of other agencies that can benefit from and contribute to the project, and disseminating lessons and products from the project. The Phase III annual project meeting will be held at the 2001 AAHE Assessment Conference.

IUPUI is the project’s host institution. It will serve as fiscal agent for the project, host a project meeting during Phase I and at other times when appropriate, and provide leadership as needed, including the leadership of its Chancellor, Gerald Bepko.

Barbara Cambridge will be project director. She is director of the Assessment Forum at AAHE and professor of English and associate dean of the faculties at IUPUI. Cambridge has both association and institution experience with setting learning outcomes, developing appropriate contributing pedagogies, and choosing assessment measures. She has worked with multiple institutions on pedagogical and assessment matters and has served on a state commission for higher education.

The project director’s duties will include

*consulting with and conducting site visits to participating institutions

*fostering networks among project participants

*planning and convening project meetings

*coordinating work of the National Advisory Board and Institutional Review Board

*promoting dissemination of lessons and outcomes from the project

Associate project director

Caitlin Anderson will be associate project director. She is research associate for the AAHE Assessment Forum. Anderson has a doctorate in higher education and experience in a major research unit and in a dean of faculites office. She has organization, Internet, and research skills essential to the project.

The associate project director’s duties will include

*conducting research pertinent to the goals of the project

*organizing meetings and campus visits

*managing budgetary matters

*providing continuous communication among project institutions, electronically and in print

*documenting activities of the project for assessment of the project

The project assistant will be responsible for the logistics of meetings, conferences, and mailings. She/he will handle the day-to-day activities of the project.

Technology development associate

The technology development associate will provide technical expertise in developing the full capacity of the Web site for information transfer and interaction among project participants (which the associate director will then manage). As importantly, this associate will research and develop the hypertextual possibilities of the institutional portfolios, working with participating universities, accrediting associations, and external users of the portfolios.

Evaluation of the project will be done during its progress and at its culmination. Several forms of evaluation will be necessary, particularly because of the multiple stakeholders involved.

1. Focus groups

A focus group will be run on each campus toward the end of each phase in order to determine if key players on the campus who are not charged to carry out the project are aware of the activities of the project. Real change is possible only with campus-wide understanding and eventual support.

2. Self report

Each institution will be asked to write a report at the end of each phase in which it identifies the benefits and any downsides of the project for its campus. After phases 1 and 2, the reports will be used to adjust activities for the subsequent phase. After phase 3, the reports will be used in building the model that can be useful for similar projects for other institutional types.

3. Critical friend

The participation of a critical friend throughout the project will enable the project to make corrections in its direction and to capture the positives. Possible critical friends would be Susan Ganter, who recently finished a stint for the National Science Foundation evaluating the large-scale effects of calculus reform and impacts of institution-wide reform efforts in the SMET disciplines (science, mathematics, engineering, and technology); Larry Braskamp, who served as acting director of the Council on Higher Education Accreditation, has published widely on assessment, and understands urban public comprehensive universities; or Cheryl Fields, associate managing editor, Point of View and Opinion, The Chronicle of Higher Education. For this project, a critical friend not currently at an institution would be particularly effective.

1. Foundation evaluation process

Pew’s evaluation unit will be helpful in suggesting ways to evaluate the strengths and weaknesses of the project in terms of Pew’s criteria for successful projects.

1. The project will generate a definition of the mission of the urban public comprehensive university that can be used in internal decision making and to inform external publics.

2. Project universities will determine a set of direct measures of learning, a set of indirect measures of learning, a set of good practices, and a description of an enabling environment for reaching learning goals at urban public comprehensive universities.

3. The project will develop a prototype of an institutional portfolio that will cultivate internal improvement and communicate to multiple publics.

4. The project will build a persuasive argument for and present evidence toward a change in accreditation practices.

5. The project will present a recommendation on the creation of a new Carnegie category.

6. The project will offer a model for similar work with other institutional types.

Bepko, Gerald. "Testimony for the National Commission on the Cost of Higher Education." October 27, 1997.

Burke, Joseph C. "Performance Funding Indicators: Concerns, Values, and Models for State Baccalaureate Institutions." Report for the Pew Charitable Trusts. 1996.

Council of the National Postsecondary Education Cooperative. "Student Outcomes Information for Policy-Making: Final Report of the NPEC Working Groups on Student Outcomes from a Policy Perspective." February 1997.

Dill, David D., William F. Massy, Pete R. Williams, and Claude M. Cook. "Accreditation and Academic Quality Assurance: Can We Get There From Here?" Change 28:5 (Sept.-Oct. 96).

Education Commission of the States. "Making Quality Count in Undergraduate Education." 1995.

_____. "Refashioning Accountability: Toward a ‘Coordinated’ System of Quality Assurance for Higher Education." May 1997.

Ewell, Peter. "Identifying Indicators of Curricular Quality" in Handbook of the Undergraduate Curriculum: A Comprehensive Guide to Purposes, Structures, Practices, and Change (San Francisco: Jossey-Bass, 1997).

Graham, Patricia Albjerg, Richard Lyman, and Martin Trow. "Accountability Study." 1995.

Legg, J. Ivan. "The Mission of a Comprehensive Urban University: The University of Memphis." Paper presented August 1997.

Shadle, Mark. "Multiple Tables of Contents: The Secret of Success with a Computerized ‘Mothership’ Portfolio." Paper presented to the New Directions in Portfolio Assessment Conference. October 2, 1992.

Shulman, Lee and Patricia Hutchings. "Fostering a Scholarship of Teaching and Learning: The Carnegie Teaching Academy." August 1, 1997.

University of Alabama at Birmingham
University of Cincinnati
Cleveland State University
Florida A & M University
Georgia State University
University of Houston
University of Illinois at Chicago
Indiana University Purdue University Indianapolis
University of Massachusetts at Boston
The University of Memphis
University of Missouri-St. Louis
University of Missouri-Kansas City
University of New Orleans
City College of New York
University of Pittsburgh
Portland State University
Temple University
The University of Toledo
Virginia Commonwealth University
Wayne State University
University of Wisconsin, Milwaukee