e-Portfolio Home :: Competencies :: Vlasta Home

competency A

competency B

competency C

competency D

competency E

competency F

competency G

competency H

competency I

competency J

competency K

competency L

competency M

competency N

Statement of Competency N

In which I discuss evaluation of programs and services on specific criteria.

---

The evaluation of projects or services is often perceived as secondary to the main purpose. However, evaluation, both as a continuous process or as a periodical activity, assures the responsiveness of the system to users’ needs and provides the base for the assessment of its effectiveness. The expectations of the effectiveness of systems and services are usually based on the patterns and trends observed through the use of traditional resources. However, we still have limited knowledge of how the use of resources and services changes with the fact that they are digital. The implementation of the on-line reference services does not directly reflect the pattern and volume observed through the similar regular in-library service.

The digitalization projects are usually conducted to provide easy access to collections. But how the digitalized collections are used and by which user groups or how they interacted with the collection interface could be ascertain only through the evaluation and usability studies. How to conduct the evaluation and which method to use would depend on the goals of the project or service and what information we would like to find.

In a digital environment, data gathering which would be the base for the evaluation could be conducted through electronic tracing of the user’s interaction with an interface and collection. We can measure the time the individual user spends logged on the system, which links or tools were used and what information was requested. This system is automatic and does not require the user’s involvement, but provides limited insight in the reasons for his actions.

Electronic questionnaires are another method of data gathering. As with electronic tracking, the electronic questionnaires have the problem that they do not provide a representative sample of the population. They are self-selective by surveying just the people that used the system and only those that were willing to answer the questions. However, the advantage of the electronic media is its ability to offer a number of ways to deliver the questionnaires. They could pop-up at particular stages of interaction or after particular services are executed. The advantage of electronically administered questions is that they could be automatically processed and analyzed.

Observing how people use and interact with the system can provide some depth to the purely numerical data gathered through electronic tracking and questionnaires. Observation usually involves observing interaction in some context, physical, social, or intellectual. Together with interviews or focus-group discussions, it provides the opportunity to actively engage users in the evaluation of the systems. Interviews and observations could be performed simultaneously, collecting the researcher user observations about the same event.

Evaluations could be carried out at the outset of the project (front-end analysis), during the development phase of the project (formative evaluation), and at the conclusion of the project (summative evaluation). Front-end analysis is often carried as the project reconnaissance and assessment of needs. Formative evaluation can help in refining the system elements, like the interface or search engine. The evaluation of the project and system could save a lot of problems and money along the way. Summative evaluation assesses the final form of the system and the user’s interaction with it. Often these kinds of evaluation are required as the part of grant reports.

The Colorado’s Historic Newspaper Collection, the case study which is my EVIDENCE 1 to this competency, is a good example of how evaluation not only provides information that helps fine tune the system, but also affects the change of standards and best practices used for the project.

In the fall of 1998, the Colorado State Library received a grant to develop an operational framework for the Colorado Digitalization Project (CDP). Colorado’s Historic Newspaper Collection (CHNC) is one of projects executed under that grant. The main rationale for the digitalization of the newspapers was the long expressed need of users to find a more efficient way to research newspapers than rolling through the microfilms or leafing through the crumbling originals. Besides the technical execution of the project that involved scanning the microfilms and converting the images into a digital database, one of the main goals of the project was to “provide the information community with usability information that may lead to improvements in access to this or another online newspaper collections” (Bibliographical Center for Research [BCR]). They included in the project outside usability consultants. In order test the usability of interface and challenges of searching 19th century documents, consultants tested groups of K-12 teachers and students at Johns Hopkins University using hybrid usability methods and task-based testing. In addition to that, the project formed four task groups:

• Language/metadata task group – which tested the effectiveness of searching, indexing and problems with the 19th century documents. The group revised the project Dublin Core Metadata Best Practices documents addressing the problems seen through the testing and also suggested appropriate revisions to the database help document.
• Interface Usability Task Group – which tested and evaluated usability of the database interface and suggested improvements based on the feedback received from user and focus groups.
• Educators/Scholars Users Group – besides the development of pedagogical tools to use historic newspapers in the classroom, this task group evaluated the use of newspapers in the K-12 classrooms.
• Public/Lifelong Lerner User Group – which was responsible for developing instructional documents for researchers and the general public.

All groups coordinated their work and shared information. Based on the task group findings and evaluation, the project realized that there is a main issue with the natural language (key-word) searches through the database of 19th century documents. It was concluded that the creation of a thesaurus specific to 19th century language is not feasible within the scope of the project’s grant. As an alternative, CHNC created extensive help documents explaining the problems in using various search terms and suggesting possible solutions.

In my EVIDENCE 2 to this competency, written as an segment assignment in the Beginning Cataloging and Classification class, I evaluate OPACs of the Riverside County Public Library System and Riverside Community College District LAMP (Library Access to Monographs and Periodicals). The assignment was to compare two catalogs operating in different library environments and powered by different software. Evaluation was to be in terms of the interface, search options and display of retrieved information. Using basic and then power search, I performed the same query on each database, and then analyzed and compared retrieved results and the searching experience. The conclusion of the paper was that, considering the different environments and users that the two libraries serve, each catalog was quite well suited for its intended purpose. However, comparing different types of libraries with different types of OPAC is like comparing apples and oranges. It is difficult to ascertain how much of the OPAC interface has to do with the vendor and how much with nature of informational needs/preferences of the patrons.

Reference:

Bibliographical Center for Research [BCR]. (2005?). Final Report Narrative. Colorado’s Historic Newspaper Collection: Final Report. Retrieved April 19, 2008, from:
http://www.bcr.org/cdp/projects/newspapers/docs/chncfinalreportreadyforpdf12-19-05.pdf