lessons

Creating electronic institutional portfolios requires the alignment of technical, analytical, evaluative, academic and graphic design resources in ways that most colleges and universities have never done before. Colleges and universities pursuing this increasingly popular assessment method will face issues regarding emerging computing platforms, needed staff skill sets, and new working relationships among organizational units. And, with new types of web-development and computer-based assessment tools becoming available at a rapid rate, development choices are becoming increasingly complex.

Confronted with these issues, the six universities participating in the Urban Universities Portfolio Project[1] (UUPP) contracted a third party consultant to perform campus-specific functional needs assessments. These assessments focused on such issues as the components of each campus’s current technology infrastructure most relevant to electronic portfolio development, the kinds of web functionality manifest in campus-based and commercially available products, the strengths and gaps in personnel and computer resources available to the project, and the longer-term implications for staffing and resource commitments to continue the development and maintenance of the portfolios after the initial project timeframe ends.

This paper highlights some of the lessons learned from this functional needs assessment. Although undertaken for a specific type of electronic portfolio project (institutional portfolios) and among a circumscribed set of institutions (6 relatively large, urban, public universities), the findings are relevant to other types of electronic portfolios (e.g., student, course, etc.) and to other types of institutions.

Defining the “Need”

The original Urban University Portfolio Project proposal[2] did not oblige participating institutions to develop their portfolios electronically. The electronic option was specified as a potential second stage of development. Participating campuses recognized at one of the first project meetings, that the dynamic and hypertext capabilities of a web platform would provide an ideal venue for prototyping institutional portfolios. It was apparent that paper formats would be too inflexible and quickly become too unwieldy given the range of evidence that would be needed to document institutional effectiveness.

Although each institution had well-developed (and rapidly evolving) campus web sites, the faculty and staff involved in the project brought to the table only a modest range of web development experience. The campus teams were composed of faculty and staff most competent in the areas of assessment, institutional research, and faculty leadership: all essential components for portfolio development. The institutional research officers had some experience with web site development, but they were not experts in this area. Moreover, the six institutions had very different information technology (IT) infrastructures and different organizational climates in general.

As the six institutions began to prototype their web-based portfolios, it became clear that there were some fundamental issues that each campus had to face. Should they hire or train staff in the institutional research or in the Provost’s office to develop the web site? Does it make sense to outsource the web development? What kinds of staffing, computing, and financial resources will be needed to continue this project when the external funding is depleted?

To address these issues, the research and technology staff of the central coordinating team proposed conducting a technological needs assessment. But discussion with the broader group placed these technological needs into a broader context. Project participants realized that the technological needs could not be considered separate from the organizational needs (what types of committee structures and organizational responsibilities were needed) and the analytical needs (what is involved with synthesizing and interpreting campus assessment results for an external audience. The task was therefore reconceptualized, as a functional needs assessment, having a combined focus on the technological, organizational, analytical and resource components that are required to develop and sustain electronic institutional portfolios.

Method

A Request for Proposal (RFP) was developed and distributed to a range of consultants. The range included consultants that focused on technology in higher education, assessment, and organizational development. Given the unique requirements of the proposal, and the limited budget, viable bids were received from only two consultants. EduTech, Incorporated was contracted to perform the assessments.

The UUPP project staff and consultants developed a protocol for site visits to each of the six campuses. The UUPP Technology Development Associate accompanied the consultants on all site visits. The consultants were required to prepare a site visit report for each campus visit, as well as an overall project report. The site visits occurred during the second year of the three-year project. As they proceeded, the campuses were continuing to evolve their development approaches. Given the rapidly evolving nature of this project, some of the issues that were most central when the assessment was defined had either become less central or had evolved into different issues.

For example, two of the six campuses moved from developing the web entirely with existing staff, to outsourcing the graphical and dynamic interface development. That is, both had come to the conclusion that it was most effective to use existing staff to generate content and not divert their attention to mastering the technological components. However, project staff from these two campuses also realized that outsourcing was only effective if the content developers had constant interaction with the contracted web-developer.

The point of this example is that the campuses were learning lessons about the functional needs of the project while the assessors were focusing on elements considered important at an earlier point in time. Thus the findings of this functional needs assessment derived as much from the campuses own experiences as from those of the consultants. In many ways, these lessons converged, and it is those points in particular that are the major focus of the summary of findings.

Findings

Overall Strategy

Virtually everyone agreed that there was less commonality among the campuses than was originally anticipated. Participants did not think this was a bad thing, or that it somehow reduced the value of bonding the six campuses together in the overall project. While the project so far has not turned up many solutions that work for everybody, the interchange of ideas has stimulated each campus to go in directions that it would not necessarily have done on its own. One example is the issue of providing routes through the Portfolio that serve the purposes of different kinds of visitors. While no two campuses have solved this problem in identical ways, the very recognition of the issue and the joint exploring of solutions helped to advance the thinking on each campus. On the other hand, several campuses have ended up ‘going the same way’ as exemplified in the common outsourcing choice made by two of the campuses as described bove

Lesson 1. It is difficult to establish a priori commonalities (e.g., using common templates or commonly prescribed development methods) in an electronic portfolio development project given the diverse nature of participants and the novelty of the electronic portfolio concept. However, commonalities can emerge from a post hoc analysis, especially if the participants share experiences as they proceed.

For example, one campus approached the project with a focus on collaboration. Early in the project, the Institutional Research office established a Faculty Advisory Committee, and involved its members deeply in deciding on the nature and content of the portfolio. This established a strong sense of partnership with the faculty and other elements of the campus and established a very positive attitude toward the portfolio project and the IR office.

In contrast, the project on another campus has been tightly focused and controlled by the IR office and its parent office under an Associate Provost for Academic Programs. The portfolio project has been closely aligned with the mission and responsibilities of these two areas. The portfolio has focused on providing data to support decision-making and communicating information about the academic program review process. In other areas, such as community involvement and campus planning, the role of the Portfolio has been mostly to assemble links to other campus resources, rather than to make a substantive contribution to their development. The newly created decision support system housed within the portfolio and the academic program review components are very visible on campus; the Portfolio itself is less recognized.

On another campus, the success of the portfolio project has made it an attractive vehicle to support additional purposes, such as planning and accreditation. While these can be valuable and appropriate roles for the portfolio, the immediate progress of the project (and the IR office) is in danger of being slowed down by the burden of taking on multiple roles.

Lesson 2. Electronic portfolios have many different potential uses and can be approached with many different strategic focuses. Articulating purpose and strategic approach early in the process are essential to a successful development effort.

Organizational Placement

Sponsorship

Where the Portfolio Project is clearly sponsored by a major office (such as a Provost or Vice President) and is clearly an important priority for that office, resources tend to be more available. Projects that have been mainly championed by a single department, such as an office of institutional research, may find themselves competing with other campus priorities for additional resources.

The portfolio project on one campus has been strongly endorsed by the Provost’s Office. This Provost signaled the importance of the project as integral to many of his own goals, but has also directed the efforts of his special assistant to the project and indicated strong support for the IR office as the unit to lead and carry out the effort.

Lesson 3. There is no substitute for the strong commitment of campus leadership. This is particularly important for an electronic portfolio project, where the concept is not well known, and a significant amount of cooperation is required from other members of the campus community.

Project Direction and Staffing

One campus’s project demonstrates the advantages of building the portfolio on the foundation of existing structures and priorities. For one thing, this has made it easier for the existing offices to agree on the nature of the content that would go into the portfolio and to divide the work up among themselves. Also, because the production of the portfolio was largely based on efforts that were already ongoing, the necessary staff resources were already in place to generate the baseline version of the portfolio. Much of the content is based on activities that the IR office has already been carrying out as part of its mainstream responsibilities and on the academic assessment activities that were already being initiated by the university’s administration. As one person summed it up, the portfolio has “given us new boxes” to put our material in.

Four of the six UUPP institutions had project directors who were organizationally affiliated separately from the major staffing office (institutional research). Furthermore, the two institutions that had closer organizational linkages between the project director and IR office were able to keep the portfolio project identified as one that was mostly independent from mainstream ‘IR’ work. This enabled the project directors to link portfolio development more closely with campus assessment processes involving faculty in the academic units. It also helped to keep the work from being seen as administrative overhead.

Lesson 4. Electronic portfolio development efforts that ‘fit’ well into existing program and administrative unit responsibilities will proceed quickly, but may then be limited by the existing programs and units’ roles in the institution. It is probably best to combine existing operations with some new ‘thinking,’ as the UUPP enabled through the Project Director/IR office linkage.

Campus Processes

Some campuses endeavored to build broad support for the Portfolio across the University, involving many groups in its design and maintenance, and achieving a high level of visibility for the project. This kind of broad acceptance can make funding easier, but it can also lead to “scope creep,” where many more good ideas get generated than can be funded or carried out. Conversely, other campus Portfolios have been “stealth” projects, deliberately kept at a low profile. These projects can be more easily controlled, but must be funded and staffed entirely by the entity that nurtures them, and may ultimately be more difficult to institutionalize.

Lesson 5. Involving a broad base of campus constituents increases the likelihood that the project will be supported and useful. At the same time it tends to create burdensome expectations. The best approach may be to incorporate a broad vision, while taking modest, incremental steps in scope and involvement.

Relationship to Campus Assessment Processes

The power of portfolio assessment emerges from the combination of its formative and evaluative components. That is the process of constructing portfolios requires the individual or unit to develop new evaluation capacities as well as to evaluate the current status of its work. The UUPP was initially conceived as both a developmental as well evaluative project. The institutions participating in the project realized at the start that this would not simply be a matter of packaging already available evidence. Rather it would require the campuses to develop assessment capacity that would ultimately yield the contents of the portfolio.

The capacity building that occurs through the development of electronic portfolios is one of the most intangible but beneficial outcomes that occurs. Several of the campuses developed specifications of student learning outcomes as a direct result of the project. Even the campuses that had well-developed processes discovered through the portfolio how to enhance those processes, particularly in their ability to synthesize results for communication to external audiences.

Lesson 6. Electronic portfolios require substantial assessment capability, but they also help produce that capability. There is a span of capability within which electronic portfolio projects can best operate. If too early, the processes may not be sufficient to sustain portfolio development. If too late, the processes may be too entrenched to benefit from the insights that portfolio development provide regarding communication to external audiences.

The Technology Domain

Processing Power

Somewhat to our surprise, the question of processing power has not been a problem at any of the UUPP campuses, at least so far. Most of the campuses developed their portfolios as pilot projects and have still not generated much traffic or put strain on the infrastructure. The projects have either used servers and other resources that were already being operated by the sponsoring department, or have made use of centralized facilities.

On the other hand, one campus that is simultaneously developing electronic student portfolios found the issue of processing power became significant in the early stages of project development. That is, processing power has far more to do with the number of portfolios than with the size of each portfolio.

Lesson 7. It doesn’t take much hardware to develop a few electronic portfolios (e.g., institutional portfolios) in great depth. However, it takes a lot to develop many portfolios, even if each one is rather small.

Where’s the Data?

The development of institutional, student, and course portfolios all benefit from the ready availability of institutional data. Institutional portfolios are likely to include statistical reports on student progress and performance, student electronic portfolios can benefit from links to the courses the student takes, and to archived samples of the students’ course work. Course portfolios also derive benefit from links to information about course enrollments.

Five of the six campuses involved in the UUPP are at various stages in the process of moving from a legacy system to a modern, integrated, and accessible institutional information systems (SCT Banner, PeopleSoft, or a combination). The sixth already has Banner in operation. This means that some of the campuses still have to work through a labor-intensive process to assemble data from the campus information system in a way that it can be presented, either statically or dynamically, as part of the portfolio. Eventually, though, all the campuses will have a powerful database to draw on, and in some cases, the development of a data warehouse is also under way. A data warehouse contains data extracted from the operational system and organized in such as way as to be useful for analysis and longitudinal study.

For institutions that are now providing only static reports in their portfolio, the coming years could bring changes in both possibilities and expectations. Once a modern campus information system is in place, it becomes natural to want to link the portfolio to a data extract, which in turn is updated regularly and automatically from the operational information system or from an institution’s data warehouse. This requires software to generate dynamic pages from the database. (Active Server Pages, Cold Fusion, PHP, and Zope’s DTML are examples that are in use or being considered on the UUPP campuses.) These links also require a dedicated database (Oracle or SQL Server, for example) and a programming tool to extract data from the campus system to the Portfolio’s own database. Although these steps become considerably easier with modern campus information system software than they were with legacy systems, there is still a considerable amount of software and tools that need to be matched and interfaced. In one of the most ambitious Portfolio software development projects, one campus has already gone so far as to create its own software to allow users of the Portfolio to design their own queries in a menu environment.

Lesson 8. The development of electronic portfolios benefits greatly from ties to the institution’s operational information systems, but these ties come at great costs, especially since so many colleges and universities are in the process of migrating their operational information systems. These ties are often best served through secondary data access systems, such as data warehouses.

Authoring Software

For designing the individual pages of electronic portfolios, and assembling them into a coherent whole, there exists a range of software tools. (Macromedia’s Dreamweaver and Microsoft FrontPage are examples of products in wide use on the six campuses.) Most recent versions of Web page editors are careful not to make proprietary or arbitrary changes to the html code, so the good news is that a number of tools can be used interchangeably to edit basic html on the same page. Most campuses are constructing the pages of the Portfolio and the navigational links by hand, rather than through automation. However, one site is prototyping a more complex system based on Zope, which includes an object database where elements of the pages can be stored and assembled dynamically when the page is called up.

Lesson 9. There is no one right answer about which tools to use. The tools selected should be ones that have solid support on the local campus, perhaps the ones that are established as official standards by the campus IT department or the Web policy committee, or the tools that have become unofficial standards because they are used by the team that is responsible for creating and maintaining the university’s official pages.

Technological Organization

For the most part, the six UUPP campuses did not involve their central IT shops in the development of institutional portfolios. Given ongoing systems migrations and other central IT priorities, it was clear that electronic portfolio development would best proceed as a somewhat self-supported effort. However, two of the campuses were more reliant on central support: the one mentioned earlier that built a decision support environment as part of the portfolio, and one that had very limited support resources for the project.

A third campus placed the web-development activities within the broader academic affairs division rather than just among project staff. This has the advantage of tying portfolio development more closely to academic planning. Also, as mentioned earlier, the campus that was simultaneously designing an electronic student portfolio platform, found that it was necessary to tap into central IT support for this more massive venture.

As already mentioned, several campuses struggled more with the question of outsourcing. The question was not so much whether to outsource, but rather what aspects could be successfully outsourced while maintaining the level of interaction necessary to ensure that the product served its purpose.

Lesson 10. Electronic portfolio development projects are best approached with independent resources that do not have to rely on central IT time and attention. Outsourcing works if and when you can find a third-party developer who listens well and is prepared to go through many iterations.

Relationship to Campus Web Site

The Portfolio projects have generally not been led by the same people who are responsible for the university’s main public Website. Each campus has had to formulate what the differences are between the Portfolio and the main university Web page, and determine how closely they should be linked. It seems likely that these questions will not be settled during the implementation phase of the Portfolio Project, but will continue to be hammered out as both sites evolve. On some campuses, there already exists an active Web project having to do with one of the Portfolio’s themes, such as community involvement or assessment of learning outcomes. In these cases, the Portfolio team has chosen simply to include this existing material by linking to it.

Lesson 11. It is hard, but necessary to draw clear lines between electronic portfolio projects and other related campus technology initiatives, such as the campus Web site, or the migration to a new operational information system. Ultimately these projects may converge, that determination should be made after the portfolio project has time to take shape in it’s own right.

Sustaining Project Development

Staffing

The UUPP teams were well conceived for the job at hand. Each campus’ Chief Academic Officer (e.g., Provost) was a member of the project team and attended the tri-annual project meetings. The Campus Project Director was typically a faculty leader who was active in assessment, but also included an institutional research director for one campus, and a vice provost at another campus. Each team had a dedicated institutional research officer (including a second IR staff member for the team where the IR director was the project director).

It became clear that the involvement of these individuals was a minimum condition for portfolio development. The functional needs assessment was particularly helpful in articulating the staffing needs necessary to keep the project going beyond the duration of the funding.

Rightsizing

Electronic portfolio projects can vary greatly in their scope and ambitiousness. However, in some general ways they need the roles outlined above (i.e., campus leadership, faculty leadership, and analytical staffing), plus technical support staff. Technical support is likely to involve a combination of project-dedicated and contracted support and at least some ties to the central IT operation. One campus that used the results of the functional needs assessment to develop a budget request for the year after funding ended, included .25 FTE for the project (faculty) leader, .25 FTE for institutional research (analytic) support, .50 FTE for a technical project coordinator, and $20,000 in web development funds. This request represents approximately $100,000 in financial resources. The campus decided to commit $50,000 to the project with further investments depending on its success.

Lesson 2. It is difficult, if not impossible, to approach electronic portfolio as a ‘marginal’ project, that is, by adding responsibilities to already busy faculty and staff. However, the resource requirements can be quite variables, leading to…

Lesson 13. Electronic portfolio development is like a gas: it will occupy any volume it is provided.

[1] Funded by the Pew Charitable Trusts, and co-sponsored by the American Association for Higher Education, the Urban Universities Portfolio Project participants are: California State University-Sacramento, Georgia State University; Indiana University Purdue University Indianapolis, Portland State University, University of Illinois-Chicago, and University of Massachusetts-Boston. Further details on the project are available at http://www.imir.iupui.edu/portfolio.

[2] Available at (http://www.imir.iupui.edu/portfolio/documents/Final.asp).