Digital Library Service Quality:  What Does It Look Like?

Martha Kyrillidou
Director, Statistics and Service Quality Programs
Association of Research Libraries
martha@arl.org

Colleen Cook
Dean of Libraries
Texas A&M University
ccook@tamu.edu

Yvonna Lincoln
Ruth Harrington Chair of Educational Leadership and
Distinguished Professor of Higher Education
Educational Administration and Human Resource Development Dept.
Texas A&M University
ysl@tamu.edu

ABSTRACT
Research in digital library evaluation has surfaced a variety of models for evaluating the performance of digital libraries.  This chapter discusses research for the user community of Science, Math, Engineering and Technology Education Digital Library (NSDL), and the research conducted in developing a protocol known as DigiQUAL®.  Our goals were to: (a) define the dimensions of digital library service quality from the perspective of the users; (b) develop a tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts; (c) identify library 'best practices' that permit generalizations across operations and development platforms; (d) enhance student learning by managing effectively user perceptions and expectations of digital library services, and; (e) identify the opportunities and barriers in establishing a digital library service quality assessment program and institutionalizing continuous product and process evaluation efforts directed toward positive and timely management of outcomes. Evaluating digital library service quality using a standardized total market survey is based on users’ perceptions and expectations. Research has demonstrated that library users perceive library service quality on different levels—they perceive library service quality holistically and simultaneously on a more detailed level that embraces the separate dimensions of empathy, place, collections, reliability, and access.  Translating and validating these dimensions into the digital library environment is a formidable challenge. As cyberinfrastructure needs are identified and new digital library environments developed, including those formed by traditional research libraries in the form of institutional repositories, the need to understand success in the digital library environment is growing.  

INTRODUCTION
In the 500 years since the invention of the printing press, libraries have been at the center of a remarkable flowering of the human intellect, serving as repositories of recorded human experience while promoting structured inquiry and critical thinking.  Libraries are fundamental to the intellectual experience and the natural creativity of the mind; they provide core services for research, teaching and learning.  What, however, defines a great library and, furthermore, a great digital library?  What constitutes excellence or effectiveness in library services, especially digital library services?  What constitutes quality digital library services in science, mathematics, engineering, and technology (SMET) education?  How does the National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL) determine whether it is delivering the best possible service for the considerable investments made in its operations?  

The problem of library service quality assessment for digital libraries is partially addressed with the establishment of the LibQUAL+® protocol at the Association of Research Libraries (ARL).  LibQUAL+® identifies and measures library service quality across institutions and stands as a useful quality-assessment tools for local planning; LibQUAL+® is a full library service quality program at ARL.  LibQUAL+® was developed to assess service quality in the traditional library environment that has been historically print-based and has been thoroughly documented in a variety of studies (Cook 2002a, 2002b, Cook and Heath 2001, Cook, Heath, Thompson 2001, 2002, 2003, Cook, Heath, Thompson and Thompson 2001, Cook and Thompson 2001, Heath, Cook, Kyrillidou and Thompson 2002, Heath, Kyrillidou, Askew 2004, Kyrillidou, Cook and Rao in press, Kyrillidou 2006, Kyrillidou, Olshen, Heath, Bonnelly and Côte 2005, Kyrillidou and Persson 2006, Kyrillidou and Heath 2004, Thompson and Cook 2002, Thompson, Cook and Heath 2000, 2001, 2003a, 2003b, Thompson, Cook and Kyrillidou 2005, 2006a, 2006b, Thompson, Cook and Thompson, 2002, Thompson, Kyrillidou and Cook, 2007a, 2007b, 2008).  With a grant from the National Science Foundation (NSF) National Science Digital Library (NSDL), we faced the challenge of assessment in an environment where all resources were digital and the emerging community of learners, developers, and teachers existed in a virtual and distributed framework of research and development activities.  

ARL proposed to address the issue of digital library service quality assessment for the NSDL by adapting the LibQUAL+® protocol for the community of faculty and students using the NSDL resources.  NSDL is highly committed to advancing the effective measurement of library service quality in order to assist libraries in continually improving the service received by students, faculty, and other library patrons much like ARL.  Since ARL member libraries and universities support extensive digital library resources, evaluating only the service quality aspects of traditional library operations is no longer sufficient in improving library service.  Digital library evaluation is a vital, yet relatively unexplored, component of research library assessment efforts.

Building, sustaining, and servicing digital library resources is a major cost for an institution.  Collectively, ARL member libraries spent more than $3.7 billion in 2005-06 on operating expenses, and costs are certain to rise.  The escalating costs of scholarly communication—especially the prices of scholarly journals and electronic databases—are among the most volatile in postsecondary education, inflating at a rate of about seven percent per annum over the past three decades.  The portion of the library materials budget spent on electronic resources is also growing rapidly, from an estimated 3.6% in 1992-93 to more than 40% in 2005-06.  ARL university libraries reported spending over $431 million on electronic resources with the majority of spending being on electronic serials and subscription services.  Eighty-three ARL libraries also reported another $15 million expended on their behalf on electronic resources through centrally funded consortia (Kyrillidou and Young, 2008).  Staff costs, an important component of digital library development and maintenance, account for a major portion of additional expenses.  

As library expenses—including digital library expenses—continue to rise, it becomes imperative for libraries to develop more effective and efficient means of allocating scarce resources to improve the quality of services. Libraries are increasingly called upon by their institutions to demonstrate value to the research, teaching, and learning enterprise in relation to the investments made in both traditional and innovative digital library services.  While ARL libraries must still meet the needs of users in all formats of information, the NSDL protocol, designed to assess the digital component of research library collections and service, could be used to evaluate the electronic-based content and access services.  With funding from NSF/NSDL, we proceeded in developing a process and tools to enable institutions to address the most important information needs of students and faculty, and to assess whether their research, teaching, and learning needs are being met effectively within the context of the NSDL digital library.  

Concurrently driving the need for new measures of library service quality is the explosion of information in both print and electronic formats.  In this information-rich era, it is vital that students acquire, through library experiences, critical information-seeking skills, and libraries must accommodate the evolving information-seeking behaviors of students and faculty in our rapidly expanding information-based economy.  Enhanced service quality assessment in digital libraries will: (1) improve intelligent retrieval of relevant information, (2) promote learner preparation for working in an information-rich society, and (3) promote scholarship and lifelong learning in both individual and collaborative settings, as well as in formal and informal modes.

This ARL project was branded as DigiQUAL® and charted new territory, complementing other measurement practices now in place in digital libraries.  The innovative model we proposed enhances the ability of libraries with significant digital components, such as the ARL member institution libraries, to engage a new and different assessment methodology that evaluates service quality from the point of view of library users in a consistent manner across digital library operations. The ultimate goal is to improve digital libraries by directing resources to those service quality issues identified by users as most important and in need of attention.  The NSDL offered an excellent starting point for developing a digital library service quality protocol across a diverse set of digital library resources and users.  Ultimately, digital library developers and maintenance agencies constantly seek new measures for outcome assessments that permit a deeper understanding of local quality issues and provide “best practices” guides for those who wish to take corrective action in a systematic way. 

The project started as adaptation of LibQUAL+® (a modification of SERVQUAL), tested in the research library environment. SERVQUAL (for SERVice QUALity) was developed for the for-profit sector in the 1980s by the marketing research group of Zeithaml, Parasuraman, and Berry (1990).  The SERVQUAL instrument in its LibQUAL+® manifestation emerged as a highly reliable and valid tool in comparing service quality across libraries and across contexts, providing a consistent framework for engaging in total market surveys. 

An example from Texas A&M University illustrates the power of this instrument to effect organizational change. In an iteration of the original survey on the Texas A&M campus, results from the Reliability of Library Catalog and User Records performance scores showed that services were falling outside the zone of tolerance.  Armed with this information, management efforts were then concentrated on learning more about that problem and developing solutions.  Through follow-up focus groups and workflow analysis, processes were re-engineered and the software for cataloging and circulation records was replaced.  Ongoing evaluation enabled refinement of the process until optimal processing was achieved. 

The ARL LibQUAL+® project gained its initial impetus from the experiences derived at Texas A&M University, where, for six years, work has taken place on translating the SERVQUAL instrument for administration in the research library environment.  Between October 1999 and March 2000, the SERVQUAL instrument was regrounded for libraries through a series of interviews at research libraries conducted by project managers.  Building upon those interviews and adhering to the integrity of the original SERVQUAL design, a series of approximately 40 questions was developed to adapt and build a new service-directed assessment instrument to measure service outcomes as perceived by consumers of library services.

The project was endorsed as an ARL pilot project at the Association’s annual meeting in October 1999.  The Texas A&M design team acquired the hardware, designed a Web form for collecting the data, and implemented the initial development which was carried on by ARL in more recent years (Cook & Heath, 1999, Kyrillidou, Cook & Rao, 2008).  The twelve participants in the pilot LibQUAL+® project were identified from a pool of more than thirty volunteers in late 1999.  From each library, survey coordinators were appointed to work with the Texas A&M/ARL design team. In January 2000, the survey coordinators and most directors from the twelve institutions met to discuss pilot requirements and timelines.  After the January 2000 meeting, most of the participating libraries were visited by the Project Managers with the purpose of building theory—qualitatively regrounding the SERVQUAL instrument through a series of interviews with faculty, graduate students, and undergraduates.  Sixty interviews were conducted on the campuses of the participating institutions to hone the questions in the SERVQUAL instrument and to assist in identifying any additional possible dimensions.  The interviews of faculty and students were transcribed and then subjected to content analysis.  The modified version of the SERVQUAL questionnaire was placed on the Web in March 2000.  Through an iterative development process LibQUAL+® has serviced more than 1,000 since 2000 across twenty-two countries and fifteen different languages. 

The qualitative work done for LibQUAL+® indicated that the concept of self-sufficiency is an important concept that needs further exploration.  Self-sufficiency appears to be a concept that is highly relevant to digital library development.  Thus the NSDL digital library assessment, DigiQUAL®, helps us explain and understand in further detail the self-sufficiency issues and their relation to service quality issues (Lincoln, 2002).  As we will see below, self-sufficiency is extended in the digital library environment as navigability and interoperability.  

We studied NSDL digital environments including CITIDEL, DLESE, ENC, iLumina, MathDL, Math Forum @ Drexel, Michigan Teacher Network, MERLOT and NEEDS.  We also studied a digital environment within a developing digital library within the University of Texas, Utopia.  These environments were selected because of prior efforts at formative assessment and evaluation.  From this initial group, DLESE and MERLOT were selected as the starting point for developing dimensions of service quality since these sites organized annual face-to-face meetings of their users.  The DigiQUAL® research team conducted user focus groups that identified 250 items describing different aspects of service quality in digital libraries.  Analysis of the items yielded twelve themes or dimensions of digital library service quality: accessibility/navigability; interoperability; DL as community for users, developers and reviewers; collection building; role of federations; copyright; resource use; evaluating collections; and DL sustainability.

EVALUATING THE NSF NATIONAL SCIENCE DIGITAL LIBRARY COLLECTIONS: CATEGORIES AND THEMES FROM MERLOT AND DLESE

Reformulating and regrounding the LibQUAL+® instrument for digital libraries was accomplished with focus group interview data from both DLESE and MERLOT participants.  The categories, themes, and issues were identified utilizing data collected at Annual Meetings of the DLESE and MERLOT groups, with users, developers, reviewers, and system administrators.  Subjected to a formal content analysis, the data began to cluster around a dozen or more themes (Lincoln, Cook and Kyrillidou, 2004, 2005).  Those themes, with samples of the data which prompted their identification, are presented below. 

Categories and Themes from the Focus Group Data
The first several categories relate to what we have termed, broadly, “design features”—that is, the design of the Web site, its initial attractiveness to users, its ease of navigation, and the critical question for its teaching potential, its interoperability.  

Web Attractiveness; Design Features

Sample items –especially those which seem the most focused, follow the category titles:

“I think the homepage is too cluttered.” (ME)
“I would want a site that is truly simplistic; there’s a lot of initial information on the homepage.” (ME)”
“I don’t think about [aesthetics] much.  Yeah, I like the newer version; I think it’s prettier.” (DE)
“The graphics (in version #1, DLESE) wasn’t anything moving around; nowhere to go.  But it’s okay [now]”
“I really appreciate no pop-ups” (DE)
“I don’t like the (MERLOT) Webpage.  There’s so much stuff going on, but there is a problem in the way it is listed.”
“And then the other way I think about it is the theory or concept behind MERLOT, which is very attractive, more so than is the actual physical site.”

Accessibility; Navigability
This category related not only to the ease of usage of the Web sites themselves, but also to the users’ perceptions of their own ability to be self-sufficient with navigating successfully to materials and objects which they desired to utilize.  Access is not only a feature of the libraries’ Web sites, it works in the opposite direction.  That is, it is also a feature which permits a user to feel efficacious in acquiring mobility throughout the site, successfully locating materials needed, and assigning parameters to limit an individual’s search and focus more tightly on specific information needed.  Accessibility and navigability enable searches to proceed more swiftly, but also enable searchers to have a greater sense of self-sufficiency in achieving their own purposes.  Some of the typical comments from focus groups include the following:
“…we see some problems with the initial accessibility.” (ME)

“Actually, the former version of DLESE I had a pretty easy time using.”

“I think it’s fairly accessible once you are accustomed to the site.” (ME)

“…there’s a lot of links, you know, like that [unusable without a username].” (ME)

“On the other hand, I do like to have an advanced search where I can specify the format and that only things with peer review [come up], so those are features that are also very important to me.” (ME)

“In terms of accessibility, I think one thing that I also find good is they open the gateway to other digital libraries.” (ME)

“Use and experience count for a lot.” (ME)

“Actually, one of the things (and this is coming from the librarian’s point of view), ... In the Version I…you could do a lot of limiting, but to do so, you had to go to ‘Advance Search.’” (DE)

“Vocabulary is an issue.” (DE)

“…You can select a group, like age groups, and I think with finding information as with traditional libraries and graphic databases is on the vocabulary, and the vocabulary isn’t clear in this library.” (DE)

“I think with anything, if you use it over and over again, it’s easier to navigate.” (ME)

“But I think another step is that the interfacing needs to be designed in such a way that the data is taken advantage of in simple, easy-to-use, and intuitive [form].” [DL]

“The more fields the person has to complete each time, the more awesome the task is to the user.” (ME)

Other Technical Aspects of the Sites 

This is a slim category which pertains directly to the reformulation and redesign of the DLESE Web site, which occurred in 2002-2003.  This comment should likely be added to other technical aspects in the previous categories, or additional, follow-up data should be collected on the design and navigability of the new and more sophisticated Web site.
“Well, I reviewed a site yesterday, because I was in one of the workshops, and I actually did ask questions about the technical aspects of the site.  I don’t know if that referred more to whether or not you could go to the links, or it meant that the actual texts, the content material, were correct or not.” (DE)

Interoperability of the Sites

“I’ve had mixed results in terms of either use [around interoperability].  I had an easy time using it, but when I let them loose to go do certain things, I had to do a lot of explaining [to the teachers with whom I was working];  I had to kind of get them into the groove, so the speak, in order to be able to use those types of things.” (DE)

“I think that the major complaint that I had…we do what we call a learning log.  It’s a little activity we do with teachers…and I ask specific questions about how they did some of the things, and they…it’s not so much that the content is beyond them, but just some of the ways of going about using [interoperable functions].” (DE)

“I think there’s a large group of educators out there that are certainly capable, knowing content, but actually using the computer, using things in that domain—it’s very difficult for them.  Again, it’s something new to them; not that they’re stupid or something like that.” (DE)

The three foregoing comments clearly relate to the self-sufficiency issue, and, like the issues of accessibility and navigability, work not only forward, with Web site design and construction, but also backward, to users’ ability to see themselves as self-sufficient and competent to navigate to materials which are of immediate use to them, especially in creating teaching modules and lesson plans.

“Yes and no.  [Interoperability] depends on the learning object that you access.” (ME)

“As part of the review process, you have to indicate whether or not the learning object is or is not inter-operable when you review the Web site.” (ME)

Social and Psychosocial Aspects of the Digital Libraries

These concepts of community and culture in the online environment may map to the “Library as Place” dimension measured by the LibQUAL+® survey instrument.  With minor modifications, many of the questions used to measure that dimension may resonate with the online community and provide a valid means of measuring the concept of virtual community and online space.  For example, the item, “A getaway for study, learning, or research” may be changed to “A gateway for study, learning, or research” in order to better suit the online environment.

Library/Digital Library as “Community”: Users, Developers, and Reviewers

We have grouped these without reference to whether the responses were from users, developers or reviewers, but we believe it’s fairly easy to see who’s contributing when you read the comments, since they are frequently self-identifying in terms of their usage or involvement.

“I think it [community] brings to mind two very important things:  1) the notion of community within and between MERLOT.. and 2) I was just kind of blown away by the fact that they (physicists) would completely ignore any kind of ‘knowledge’ that we obviously have in the library community on how things should be done, and they built this thing along the lines of how they actually think.” (MER)

“[In building the physics part of the digital library], it’s the community base, because it’s the way they thing, because now you get to CINDY and you realize CINDY is not concept-based.  Bingo, it’s time, space and matter!  It’s really cool…”

“I’m beginning to understand now that this whole notion of community is incredibly important—probably far more important than us coming up with any ‘grand schema’ if you will.” (MER)

“At my college, we have an undergraduate library and lots of other libraries.  It is busy there.  It is a place that people can [use to] study and come together.”

“But notice what we’re saying:  That the library as a resource area where you went for stored knowledge has shifted suddenly now to a place where we look at community and social community which many librarians would say, “Sure, it’s always been that way.”  But perhaps not to the extent that we’re talking about now.”

“There’s a certain…what I kind of noticed is that DLESE has its own certain language and sort of approach to all sorts of things, and I’m wondering how an ‘outsider’ would approach it…”

“It’s a multi-step process.  I would say there’s—you’ve got researchers producing the data;  you’ve got the digital library developers producing tools, portals and stuff like that, maybe the meta-data material;  You’ve got educational material developers who are going ahead and building these lesson plans, along with teachers who may be building these lesson plans and the materials around it.  And it may wrap all the way around…”

“And that’s a perfect example of something that DLESE should be, a circle where it goes around.  Now, students have produced data that teachers have mentored the students into producing data…[And that addresses] also the issue of community, because you know that you are collecting data in California and someone in New York is using the same protocols and collecting data there, and you compare your points.”

“Offers a wonderful sense of community.”

“One of the big advantages, or the benefits, I get from MERLOT is the community and the collaborative nature of the organization as a whole.”

“But we need them [feedback forms on sites], because we are a community-driven resource.” (DL)

“And if we are a community-driven resource, we need to hear from the community loud and often.” (DL)

“That’s why it’s called a community, because you’re contributing to the community” (DL)

One of the issues to be addressed at some later point is the larger issue of “community” and what counts for a community.  We already know that the term “community” no longer means the same as it did fifty, or even twenty-five, years ago.  But the meaning of community here is far different from its original meaning.  It is intimately wrapped up in a project which creates a common “product” or service (the blurring of product and service aspects is another interesting aspect of the digital libraries)—a digital library and its archives—but its “members,” particularly its users, may be unknown to each other, and indeed, many of them will never meet.  Nor will some of them ever even “chat” with each other.  They provide input, learning objects, materials, data, and they withdraw from this common source materials, data, or learning objects which they themselves might need.  But some of this activity is done relatively anonymously, and with no contact other than the common resource which is being both built and drawn from.  The idea of “community” is therefore related to the process of building and using a common project (or service, perhaps?).  Contributors and users alike see themselves as drawing from a project which has meaning larger than them.  This entire issue may be related to more symbolic meanings and roles of the library, and its metamorphoses through the ages (Lincoln, 2002). 

Collection-building:  How It’s Done, How It Will Be Done in the Future

There is great concern about how the collections in the digital libraries are built and maintained, that is, updated, renewed, and new scientific findings (as well as teaching units) included and integrated.  There is no small amount of concern about ongoing funding, as well as simple manpower to update, vet, referee, and manage submissions, as well.  While thousands of users appreciate having the digital libraries as resources, it is quite clear that far fewer are willing to be involved in the time intensive labor necessary to build and maintain them.  In fact, Marianna Torgovnick (September 5, 2008,) suggests that archives of all sorts, particularly digital, microfilm and microfiche, are far more fragile and “ephemeral” (p. B14) than we would like to believe.  Torgovnick suggests that “the idea of the archive, independent of any specific topic, has itself become an object of obsession” (p. B14), simply because “…they depend on living beings to animate them” (p. B15).  Most assuredly, the “living beings” who animate the NSDL’s digital libraries must expend enormous amounts of time, energy and data collection and object-development to prevent the digital archives from falling prey to “link-rot” themselves.  And funding continues to be a source of anxiety and tension within the multiple communities of users, developers and reviewers.

Role of “Federation”

“So I did a search in MERLOT, and I found a few things that I think might be useful for developing the content that we’re putting together; but, for the most part, I’m hoping that our institution will be a provider of information.”

“…but there’s a lot of null searches that should be avoided, because we need to populate the library…” (DL)

“I’m on the editorial board and spend a lot of my time on MERLOT building the archive.”

“[Populating] means identifying areas that are being asked for that we’re not able to come up with.”

“That populating means also identifying an individual or set of individuals who can make it richer and richer in terms of the material that are there…” (DL)

“So, people that have been or were already on board are thinking ‘contribute!’  So…if “oceans” are weak, you call NOAA or somebody to get them on board.”

“In a similar situation, I’m on the editorial board of History and spend a lot of time just cruising history sites to fill in the gaps…”

“One of the things is to populate the library.  Right now, we can tell you what resources are in the library, and we have someone who is doing collection assessment so that we can--.  And so we’re identifying that these resources should be increased in the library before asking for them.” (DL)

“Collection building…we will forever be in a collection-building phase as far as I can tell.” (DL)

“It was previously in its rapid growth stage and trying to figure out who we are and how to tell it to people.  And now we’re into what don’t we have and why can’t people find what they want.” (DL)

“So, the collection is uneven, and it’s not clear to begin with to the user that it’s uneven.” (DL)

“Collections can come into the library with a reviewed status if they choose to.  In order to do that, they have to—like DWEL did—they created this very rigorous review program that they have to outline and adopt, and they came forth with their collection and review document.  And they have to say how they reviewed all the resources in their library for scientific accuracy, for use in a classroom, for standards…”

“That’s why we have to start looking at the collection and say, “Ok, guys, we need to go out and to collect in these [low-density collections] areas”—and that’s happened. (DL)

Federation’s possibilities:

“We heard this morning at the general session about a federated search, and I really resonated with that because as good as MERLOT is, it’s not going to be able to solve it on its own.  It’s got to be part of a federation…But that could help, if the digital libraries framework could in some way support a federated search so that you could have the same good feelings in a federated search that you have in the search that you do with MERLOT, it would be wonderful.”

“And you can tell they’re starting to fuse and they’re mingling.  It’s, like, my collection is already in two of the digital libraries.  I’m sure that there are others that are in more than one.  And so the issue is that they can federate between them.  The more they can interchange what they have inside them, the more it will become like one digital library.”  (DL)

With respect to this last comment, it is becoming clear that that there are many, many digital libraries out there, slowly becoming integrated (federated) such that it is difficult to tell just how many digital libraries actually exist, or how they might be accessed.  As better and more comprehensive meta-data are created, the possibilities for federation seem likelier.

Problems Particular to Libraries and Digital Libraries

Librarians everywhere struggle with the question of meta-data.  Even as new meta-data archives are being built, knowledge is expanding so rapidly that even the newest meta-data databases are struggling to keep up.    
Meta-data:  Their Role  

Meta-data may be a subset of the category above, just as it is likely a subset of the category “community.”  Many of the items here relate to the use of meta-data across collections, for instance, which apparently becomes a problem when collections and libraries are being/becoming “federated.”   

“So, we’ve actually felt the need for a discipline-specific page that introduces information for the first time about how things are structured, why we’ve made certain choices, how they might find things within history.” (MER)

“The way around meta-data issues, it is ‘cross-walking.’” (DL)

“Basically, if I’ve got a meta-data standard, and someone else has a meta-data standard, you can map onto them.  It may not be perfect mapping, but [cross-walking means, for example, that you can move one collection into the format of another collection]….And so, the question then is, how to federate all these digital libraries so material usually can be discovered through all of them.” (DL)
 
Please notice the whole idea of metadata just transposed itself into the federation issue here.  Mapping knowledge is a project which is not likely to be complete in our lifetimes, and metadata will continue to be both the salvation and the nemesis of digital libraries and their users for the foreseeable future.  
“My impression is that users want to have resources described in a way that anticipated the things that they want in that resources, whether it is a Web site or a learning object, how granular it is [that is, is it an image, or is it an entire Web site].”

“I think that it is worth the investment of putting educational descriptors on the resources and getting educational description of the resources that help end users find and use the learning objects or resources.” (MER)

“I want the user in a seamless sort of way to be able to find [what he needs], and in order for them to find that, that resource has to be described on the back end with those kinds of things [descriptors].” (MER)

“A lot of collections are using very minimal meta-data—contributor, date and that’s it.  And if you have richer meta-data, richer than the double-core metadata.  And it will be much easier, for example, if you’re using the learning object meta-data for end users to find what they want quickly.  And so those would be in the catalogue records, not attached to the actual resources.”

“…and the critical thing is we’re still developing a control vocabulary, a vocabulary for the library.” (DL)

Copyright Issues; Ownership of Materials

Plagiarism and the Digital Libraries. 
Just as plagiarism has remained a persistent issue with other forms of intellectual property, plagiarism has been an even more difficult issue with digital materials, particularly those which are unsigned and unattributed.  Frequently, the creators of pages and objects are credited, but occasionally, individuals create materials which are unsigned.  What counts as plagiarism in this context?  That is a question which has yet to be settled, although the comments below suggest some answers.

“Who owns the content?  Who owns what’s up on MERLOT?  Ans: People who created their pages.”

“The people who create it…have the ownership.” (MER)

“Very early on, people said MERLOT wouldn’t work because of the copyright issue, and MERLOT said, we’re not even going to deal with it; we’re just going to refer to it, we’re never going to have it ourselves.”

“Just like the Smithsonian, we own it, it’s ours, belongs to the—users.” (DL)

“I think in terms of ownership, I think it’s very difficult.  Certainly, if there are copyright statements available, there is clear ownership.  But oftentimes, there is not any copyright statement.  Then we have things like images from NASA that are public domain.  So, I don’t think that we should assume that it belongs to the authors.  I think those things are out there to access and use.” (MER)

“But it [the copyright/ownership issue] does suggest something about ownership and about the way we now view knowledge as a multi-person constructed set of instructions.” (MER)  

This issue seems important to us.  We perhaps should especially look at users’ views of what knowledge is, given the “community” nature of knowledge-building that is going on, and the multi-site construction of that knowledge.
Issues with Submissions to Digital Libraries (especially of “learning objects”) and how faculty members get credit in the promotion and tenure process.

“Who owns the content?  You own your own content, but we own the meta-data.  I mean, we can have the meta-data…The meta-data can be harvested by NSF.” (DL)

“…let’s say you have a TT plan, you have to provide us the meta-data for that resource.” (DL)

“Or you can provide a resource and ask DLESE to do the meta-data link to a single resource.”

“[Or] you can create your own meta-data record, a cataloging tool.” (DL)

How Digital Library Resources Are Used

It is worthwhile here to note that this section of data units is the second largest in the analyses.  While sheer numbers do not “count” in the same way as they might in more experimental inquiry, nevertheless, data unit count is one measure of salience in interviewing and other kinds of data collection which lead to content analyses.  There are, of course, analytic and conceptual problems with this significance estimate; one might be, for instance, inadequate or inept analysis.  Another might be unsophisticated questioning.  But we feel relatively confident when the transcripts are compared with notes which were taken in the various focus group sessions by professional ARL staff. 

“It’s also an easy way for me to send people to those sites without my having to remember what the URL is.” (MER)

“I may not use a particular site the way it was constructed as an assignment in class;  but I get ideas of how I might work an assignment on a topic…”

“And what I found is that they [teachers] really sought the animations, the quick things that they could take into the classrooms 10 minutes before class.”

“Their [the teachers] feedback—and I can identify with this—was the quick and easy—the search engines helped them find information they could use.” (MER)

“I have the most cluttered desk that you can imagine, and this is a hell of a good way to keep it organized and simply in one spot [putting the URLs and materials on MERLOT]  so that I can do exactly the same thing…”

“…if a faculty member realizes that stuff is going to come down and they still want to have it, they can negotiate to have that put somewhere else to have some permanency.”  (MER)

“So I use both a search and a harvest.  I have to tell you that I also harvest stuff.  There are tools out there—some of them very specialized, some of them not so specialized….And I must tell you that I shamelessly harvest stuff, and I’m sure I’m committing all kinds of copyright violations on a daily basis—I’ll confess to doing that!  But I do that because I’m fearful that the stuff will sometimes disappear and go away, and so I harvest that.  So I’m both a searcher and a harvester.  And that’s how I use MERLOT.”

“It has some levels of cumbersomeness in getting to the subject matter.” (MER)

“Rabbit trails are easy with MERLOT.”

“I don’t think MERLOT ever thought it was their responsibility to create and develop the learning objects.  It simply wanted to make available the learning objects and to give credit—scholarly and academic credit—to the developers, the authors.”

“But my point is that’s why we’re so often looking for those delightful learning objects that are—and you heard me say it before we started—the ‘sweet spot.’…The shorter ones that you pick to plug in?  Yeah, the ‘sweet spot.’” (MER)

“My collection has 75 different countries…and, 11,000 users over the last year.  And I’d say a good 15% of them are foreign users, at least.” (DL)

“Like, I know the San Diego super computer has 20% of our users, and I have no clue what they’re using it for yet.” (DL)

“I use MERLOT basically to show all my faculty how to find online course objects that they can use in conjunction with their regular classroom instruction.”

“I have used, searched it for information.” (DL)

As readers can see, the digital libraries are used for all the purposes originally planned for them, and more.  Some appear to contain quite active, dynamic data bases which are being created daily (daily atmospheric data, for instance, as a cumulative record), other things are mobile but not dynamic (simulations), and still others are fairly static, but nevertheless, extremely useful to some groups (e.g., middle school teachers).

Evaluating the Digital Library Collections.  

This is a set of suggestions from the focus groups members (both DLESE and MERLOT) about what should be evaluated, who should evaluate, how it should be evaluated, how and who should be vetting the collection, and the like.  Below are a set of representative comments.  This was by far the largest subset of categories, although in the analysis, we have the set divided into this one (front of the stack), and what seemed to us to be a subset, “Intellectual Issues with Digital Libraries {Teaching}.”  
“If you’re wanting general advice about evaluating, I’d say, first is, what’s your policies and all that stuff.  ‘Cause there’s a lot of policy issues that go into this.” (DL)

Some comment needs to be made about the comment above, because it has become clear that each of the NSDL digital libraries is governed by its own set of user- and developer-created policies.  The policies for usage, contributions, refereeing and other matters will clearly affect the manner in which the individual libraries are evaluated.  This may be less true when the evaluating individual was a contributor to policy development, but more true when the end user had no role in developing policies which now govern the individual library.

Issue of “dead links”, “link rot”, or “site decay”—who’s responsible for cleaning that up?

“…If I’m translating you right, there are really two levels at which stuff gets reviewed.  One is for the accuracy of the data that are on there, for the actual scientific information.  The other [level at which materials on DLESE get reviewed] is pedagogical utility and facility.”

“You figure that in a peer-reviewed process, if the image were bad science, that the image wouldn’t get up there.  So that, I think the values of being able to see the review is that you can have things that can be useful but that still do have some [l]imitations.  I mean, you may be more aware of what those limitations are.” (MER)

“But there are things where people have put a page and they haven’t touched that page in 3 or 4 years to review it, and it just sits there.  Some decision about when do you revisit these thing to see how current they are…” (MER)

“We don’t have enough in some areas.” (MER)

“From users, you would want to go ahead and ask—user testing, what kind of user testing do they do.” (DL)

“How do they gain their quality of material, the pedagogical quality.” (DL)

Problems

“Mainly, the broken links are the problem for me…”

“These teachers are desperate for this….not just [high quality] teaching material, they want data—scientific data.  They way, they say it’s more exciting if it’s real time…”

“The one thing I’ve discovered with mine, just having real-time data isn’t enough….It’s not just the data, you have to have the documentation;  and it doesn’t have to be a full lesson plan.  It’s just enough so they can know the lesson plan.” (DL)

“I guess because there’s so much information there, it wasn’t as intuitive as a lot of Web sites that I’ve been to before, and I went to a lot…” (DL)

“I went to a lot of Web sites looking for things to engage the students, and I guess I had a problem just figuring out how to narrow and search it.” (DL)

“But I have a frustration with the Web site.  Giving feedback is limited; it’s limited to the front page.” (DL)

“But honestly that is an issue with me.  I think it’s a quality issue within the library…when you like put up your list and you think you are getting all the right stuff and it puts you on a bad site, you want to be able to say, ‘I really question this site.  Could you check it out and make sure that it meets the criteria for DLESE?.’”  

Sustainability of Digital Libraries; Issues of Whether or Not They Will Continue to Be Funded, and Related Concerns.

Users, developers, contributors, and other interested parties are deeply concerned regarding whether or not NSF will continue to fund the national science digital libraries.  The tone in which focus group members spoke of this issue suggests it is a grave one for them.  Indeed, many have apparently come to believe the digital libraries are a critical national resource in the information age, and cannot imagine what their scientific life would be like without them.  Other members of the focus groups point to the knowledge that a portion of usage comes from international parties, and they construct this usage as a critical effort at scientific sharing within the international scientific communities that are utilizing the resources, especially the real-time data.  

Members of the focus groups were quite intense on querying whether or not members of the research and evaluation team would learn of the National Science Foundation’s strategy for sustaining and maintaining these libraries over the long haul, and many expressed concerned that there might be no other agency, corporation, or body which could or would assume responsibility if the Federal government were to cease its funding.  Although none of our interviewees said so directly, they want NSF to understand that these critical resources must remain a priority if the nation is to remain scientifically competitive, and retain its scientific leadership in the larger international community.

“I would ask [NSF] for the strategy of sustainability…because when it comes down to it, if it can’t be sustained, then it really doesn’t matter.” (DL)

“When you talk about sustainability…are you talking about how are we going to keep the federal government funding it?  Or are you talking about how do we keep growing the collection and the community?”  “Both.” (DL)

“And so, it was part of the proposal process you write in, what’s your sustainability path?  We get enough users, DOE wants to fund it then.  We have probably 8 to 10 times the number of users that DOE has for its data, and we are having trouble getting funded by it.” (DL)

“…There’s a reason…it’s partially because of good will, good financial politics.  If people know about it, people like it, they’re interested, it’s more likely you’ll get money from Congress.” [DL]

“What about leisure use?  What about wanting just to know something—instead of being the Monday New York Times, you go to DLESE to find out something cool and new about what’s going on.  So, I really—I mean, it may link to sustainability as you mentioned—but really make it a tool that every person…”  “And that would help in sustainability…”

“…and, you know, I’m not so sure how long MERLOT is going to be able to sustain itself.” (MER)

Other themes which emerged, which appeared to possess less salience for interviewees, included:  1) student experiences with and in libraries and digital libraries; and 2) textbooks and textbook usages versus using digital libraries.  Two other categories also emerged in the categorization process:  “Scientific literacy,” which refers to the role of the digital libraries in creating scientific literacy, and in creating a “market” for scientific data, and “The Nature of MERLOT:  Referatory rather than Repository.”  The first of the two might be “recognizable” as something which should be included in the new Web LibQUAL+® for digital libraries; but we are not certain the second will.  

Scientific literacy as a category seemed powerful to some interviewees.  As some of the foregoing data points indicated, some users and developers seem to feel that having the digital libraries open to anyone with curiosity meant that individuals might find their appetites whetted for more scientific information—genuine data from scientific findings.  Several participants commented that creating scientific literacy in ordinary people was a way of increasing general sophistication and desire for more scientific exploration; in other words, the digital libraries might help to create a more powerful market for scientific work itself. 
 
The research and evaluation team took a first shot at “mapping” the concepts to try understanding how the topical and category areas related to each other.  Doing so, we felt, would create a more integrated sense of what concerns we should focus on in the evaluation effort for NSF.  The category map, in slightly different terms, appears below, showing some of the implied relationships between categories in respondents’ minds.

[Figure 1 about here]

Clearly, while we believe we received valid and robust data from focus group respondents, we will be attempting to refine, clarify and augment those data as we refine DigiQUAL®, for the online survey of users, developers, contributors and reviewers for the digital libraries, and thus we also seek input from audiences in venues in addition to the focus groups.  Attendees at presentation sessions, readers of this paper, and other interested parties who have substantive comments to make on categories, potential items for a Web-based survey, and/or issues and concerns surrounding the services which digital libraries offer are invited to submit comments, suggestions, or other helpful inputs to paper authors.

DEVELOPING SURVEY ITEMS
Using the actual wording of the users in the focus groups we constructed survey items from a grounded theory perspective.  After identifying survey items and mapping them to themes of digital library service, we reduced the number of items to 180 based on perceived relevancy to the NSDL digital libraries profiled in earlier activities. The items were then tested in a pilot study of eight digital libraries during 2004-05 and confirmed the emerging model of a digital library as depicted in Figure 1 and its evaluation components.

The following digital libraries tested DigiQUAL® by August 2005: CSERD, Illumina, Math Forum, MERLOT, NEEDS, NSDL Core Integration, and Utopia.  Math Forum collected the largest number of surveys (775 completed surveys) (Kyrillidou and Giersch, 2005).  The rest of the sites had only very small number of surveys returned.  We are trying to understand how we can increase response rates to a total market survey like DigiQUAL® in the Web environment.  It seems that an optional survey link is not working very well in this environment and that more compelling reasons need to be implementing for collecting robust and meaningful data. Based on the pilot implementation, we can draw some conclusions about issues to consider when conducting evaluations across digital libraries, specifically in the context of NSDL. Our major findings address those lessons learned.
FINDINGS

1. Defining the dimensions of digital library service quality from the perspective of the users

While some of the items from the user focus groups overlap with items that are used in the LibQUAL+® protocol, the twelve themes are sufficiently different that the research team recommends that the DigiQUAL® protocol continue to be developed in the context of digital libraries.  A question for further study is whether there will be any difference in the relevance of themes if they are applied to digital collections developed in academic libraries rather than collections of born-digital items and services.

2. Creating a tool for measuring user perceptions and expectations of digital library service quality across digital library contexts 

2.a. The vocabulary and content of the items developed during the focus group had some interesting differences across the pilot sites.  All 180 items are not applicable across all sites.  To address this during pilot testing, several pilot sites were consulted to provide input about the items and they had the option of actually selecting the maximum number of items that would be applicable to their sites from the 180 items tested. 

2.b. During the pilot testing of the survey form, participating sites requested that more, rather than fewer, demographic items should be included since this was often the first opportunity sites had to gather feedback about and from their users. The researchers had to balance this request with developing a survey form of appropriate length. However, this, along with other findings below, suggests that a qualifying process may need to precede applying the DigiQUAL® protocol at a site.  For example, a qualifying process might first identify the users and their demographics and then approach them asking questions about digital library service quality.

2.c.  DigiQUAL® is technically scalable. Having a centralized, secure Web site where participants can manage how their site’s surveys are developed and modified frees ARL’s developer from having to create a customized survey for every digital library, and it reduces technical and organizational barriers to digital library developers deploying DigiQUAL®.

3. Establishing a digital library service quality assessment program as an integral part of the library service quality assessment program at the Association of Research Libraries

ARL member libraries are receptive to the idea of using DigiQUAL® in the context of their locally-created digital collections.  Implementing DigiQUAL® in additional digital environments provides opportunities for further validating the survey items.  This work is closely linked with the Measuring the Impact of Networked Electronic Resources (MINES for Libraries®) and linkages are explored (see related chapter in this volume by Franklin, Kyrillidou, and Plum).  It also relates to efforts to develop e-metrics for libraries (Kyrillidou and Giersch, 2004, Shim, McClure, Fraser and Bertot, 2001).

4. Identifying digital library "best practices" that permit generalizations across operations and development platforms.

Digital libraries must have a critical mass of users committed and willing to fill in survey data before conducting evaluation activities such as a total market survey.  They will have to rely in contextualized methods for actionable results in the meantime.  Conducting evaluations with digital libraries was challenging because most sites do not know who their users were or how to contact users directly.  The researchers attribute this to the short-term, project-based nature of digital library development (e.g., little time or budget to conduct extensive demographic evaluations) and to the education and research focus of many sites, especially those developed for users under eighteen (e.g., no login required).  Not knowing approximate demographics for sites’ user groups made it impossible to gauge response rates and to assess whether notification measures worked.  In most cases during pilot testing, the survey response rate was low due to a lack of a critical mass of users, so the DigiQUAL® survey yielded little actionable data.  In all cases, it was difficult to interpret survey responses as representatives of a site’s user base because there was no indication that all of the users had been notified of the survey.  For subsequent DigiQUAL® implementations, the researchers recommend developing a profile of a digital library that would, among other things, identify if there are enough users to provide a significant number of responses.  The researchers recommend that when evaluating digital libraries a mixed-methods approach be applied that combines quantitative and qualitative (interviews, focus groups) methods.

5. Institutionalizing continuous product and process evaluation efforts directed toward positive and timely management of outcomes

Digital libraries are reluctant to share individual results of evaluation efforts.  During the process of implementing the NSF funded DigiQUAL® study, researchers learned that participants were not interested in sharing results about their sites.  While the digital library community values a collaborative approach towards accomplishing project work, projects are faced with a competitive funding structure that provides no incentive for sharing data.

The depth and breadth of commitment by research libraries is indicative of the urgency of the need for outcome evaluation in libraries and is a reflection of the enthusiasm of the postsecondary library community for alternative measures. Adapting the LibQUAL+® for the digital library community remains a necessary and important next step.  The digital library environment continues to morph and developments such as cyberinfrastructure, cyberlearning and e-science (Tenopir and King, 2004, Borgman, 2007, Association of Research Libraries/Coalition for Networked Information, 2008) will influence further our notions of what a digital library is.  

There is a danger of allowing the means to become the end in the form of a technological determinism—rigorous understanding of the disciplinary transformations and the challenges of transitioning into digital and less print resources is helping libraries engage in systematic assessment methods.  Assessment and evaluation coupled with responsible development will help libraries survive and thrive into the new century.

References
Association of Research Libraries/Coalition for Networked Information (2008). “Reinventing Science Librarianship: Models for the Future” Proceedings from the ARL/CNI Fall Forum, October 16-17, 2008: http://www.arl.org/resources/pubs/fallforumproceedings/forum08proceedings.shtml.
Borgman, C. (2007). Scholarship in the Digital Age (Boston: MIT Press).
Cook, C. (Ed.). (2002a). The maturation of assessment in academic libraries: The role of LibQUAL+TM (special issue). Performance Measurement and Metrics, 3(2), 34-112.
Cook, C.C. (2002b). A mixed-methods approach to the identification and measurement of academic library service quality constructs: LibQUAL+TM. (Doctoral dissertation, Texas A&M University, 2001). Dissertation Abstracts International, 62, 2295A. (University Microfilms No. AAT3020024).
Cook, C., & Heath, F. (2001). Users' perceptions of library service quality: A "LibQUAL+TM" qualitative study. Library Trends, 49, 548-584.
Cook, C., Heath, F., & Thompson, B. (2001). Users' hierarchical perspectives on library service quality: A "LibQUAL+TM" study. College and Research Libraries, 62, 147-153.
Cook, C., Heath, F. & Thompson, B. (2002). Score norms for improving library service quality: A LibQUAL+TM study. portal: Libraries and the Academy, 2, 13-26.
Cook, C., Heath, F. & Thompson, B. (2003). "Zones of tolerance" in perceptions of library service quality: A LibQUAL+TM study. portal: Libraries and the Academy, 3, 113-123.
Cook, C., Heath, F., Thompson, R.L. & Thompson, B. (2001). Score reliability in Web- or Internet-based surveys: Unnumbered graphic rating scales versus Likert-type scales. Educational and Psychological Measurement, 61, 697-706.
Cook, C., & Thompson, B. (2001). Psychometric properties of scores from the Web-based LibQUAL+TM study of perceptions of library service quality. Library Trends, 49, 585-604.
Heath, F., Cook, C., Kyrillidou, M., & Thompson, B. (2002). ARL Index and other validity correlates of LibQUAL+TM scores. portal: Libraries and the Academy, 2, 27-42.
Heath, F., Kyrillidou, M., & Askew, C. (Eds.) (2004). Libraries act on their LibQUAL+TM findings: From data to action (special issue). Journal of Library Administration, 40, 1-239.
Kyrillidou, M. (2006). Service Quality: A Perceived Outcome for Libraries. In Revisiting Outcomes Assessment in Higher Education, ed. P. Hernon, R. E. Dungan, and C. Schwartz, (Westport and London: Libraries Unlimited), 331-366. 

Kyrillidou, M., Cook, C., & Rao, S. Shyam Sunder.  (in press). Measuring Library Service Quality and LibQUAL+® in Academic Libraries Research Perspectives, 2 (Chicago, IL: ALA/ACRL, in publication) 
Kyrillidou, M. and Giersch, S. (2004) Qualitative analysis of ARL E-metrics participant feedback about the evolution of measures for networked electronic resources. Library Quarterly, 74(4). http://www.journals.uchicago.edu/cgi-bin/resolve?LQ740402PDF. 

Kyrillidou, M. and Giersch, S. (2005) The DigiQUAL Protocol for Digital Library Evaluation, JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. http://old.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf. 

Kyrillidou, M., Olshen, T., Heath, F., Bonnelly, C. & Côte, J. (2005) La mise en œuvre interculturelle de LibQUAL+MC Le cas du français. BBF Paris, t.50(05), 48-55.

Kyrillidou, M., and Persson, A. (2006). The New Library User in Sweden: A LibQUAL+™ Study at Lund University. Performance Measurement and Metrics, 1, 45–53.
Kyrillidou, M. and Heath, F. (2004). The Starving Research Library User: Relationships Between Library Institutional Characteristics and Spring 2002 LibQUAL+™ Scores. In Libraries Act on Their LibQUAL+™ Findings: From Data to Action, eds. F. M. Heath, M. Kyrillidou, and C. A. Askew (New York: Haworth Press), 1-11; Co-published simultaneously in Journal of Library Administration, 40, 1–11.

Kyrillidou, M. and Young, M. (2008). ARL Statistics 2005-06. (Washington DC: Association of Research Libraries).

Lincoln, Yvonna S. 2002. Insights into Library Services and Users from Qualitative Research. Library & Information Science Research, 24(1), 3-16.
Liconln, Y., Cook, C., and Kyrillidou, M. (2004) “Evaluating the NSF National Science Digital Library Collections,” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf.

Lincoln, Y., Cook, C. and Kyrillidou, M. (2005) “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSL Digital Libraries and LibQUAL+™ Data Bases”
7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona 6-8, July, 2005.  http://old.libqual.org/documents/admin/ISKO.PDF.

Tenopir, C., and King, D. W. (2004). Communication Patterns of Engineers. (Hoboken, NJ: John Wiley and Sons, Inc.).

Thompson, B., & Cook, C. (2002). Stability of the reliability of LibQUAL+TM scores: A "Reliability Generalization" meta-analysis study. Educational and Psychological Measurement, 62, 735-743.
Thompson, B., Cook, C., & Heath, F. (2000). The LibQUAL+TM gap measurement model: The bad, the ugly, and the good of gap measurement. Performance Measurement and Metrics, 1, 165-178.
Thompson, B., Cook, C., & Heath, F. (2001). How many dimensions does it take to measure users' perceptions of libraries?: A "LibQUAL+TM" study. portal: Libraries and the Academy, 1, 129-138.
Thompson, B., Cook, C., & Heath, F. (2003a). Structure of perceptions of service quality in libraries: A LibQUAL+TM study. Structural Equation Modeling, 10, 456-464.
Thompson, B., Cook, C. & Heath, F. (2003b). Two short forms of the LibQUAL+TM survey assessing users' perceptions of library service quality. Library Quarterly, 73, 453-465.
Thompson, B., Cook, C., & Kyrillidou, M. (2005). Concurrent validity of LibQUAL+TM scores: What do LibQUAL+TM scores measure? Journal of Academic Librarianship, 31, 517-522.
Thompson, B., Cook, C., & Kyrillidou, M. (2006a). Stability of library service quality benchmarking norms across time and cohorts: A LibQUAL+TM study. Paper presented at the Asia-Pacific Conference of Library and Information Education and Practice, Singapore. http://www.coe.tamu.edu/~bthompson/libq2005.htm.
Thompson, B., Cook, C., & Kyrillidou, M. (2006b). Using localized survey items to augment standardized benchmarking measures: A LibQUAL+TM study. portal: Libraries and the Academy, 6, 219-230.
Thompson, B., Cook, C., & Thompson, R.L. (2002). Reliability and structure of LibQUAL+TM scores: Measuring perceived library service quality. portal: Libraries and the Academy, 2, 3-12.
Thompson, B., Kyrillidou, M., & Cook, C. (2007a). On-premises library versus GoogleTM-like information gateway usage patterns: A LibQUAL+® study. portal: Libraries and the Academy, 7, 463-480.
Thompson, B., Kyrillidou, M., & Cook, C. (2007b). User library service expectations in health science versus other settings: A LibQUAL+® study. Health Information and Libraries Journal. 24(Supplement 1), 38-45.
Thompson, B., Kyrillidou, M., & Cook, C. (2008). Library users' service desires: A LibQUAL+® study. Library Quarterly, 1, 1-18.
Torgovnik, M. (2008). File under fleeting: Archives are more ephemeral than they seem. The Chronicle Review, September 5, pp. B14-16.
Shim, W., McClure, C. R., Fraser B., and Bertot, J. C. (2001). Data Collection Manual for Academic and Research Library Network Statistics and Performance Measures (Washington, DC:  Association of Research Libraries). http://www.arl.org/stats/initiatives/emetrics/index.shtml.

Zeithaml, V.A., Parasuraman, A., & Berry, L.L. (1990). Delivering quality service: Balancing customer perceptions and expectations. (New York: Free Press).