Scholarly Habits and Digital Resources: Observations from a User Survey

Mellon Survey Report
January 11, 1999

Introduction

This report describes a survey conducted by the Brown University Women Writers Project in the fall of 1997, as part of a research project funded by The Andrew W. Mellon Foundation. It was originally prepared for presentation at the Digital Resources in the Humanities Conference (Glasgow, September 1998). This expanded draft draws on preliminary analysis of the survey data.

Within the overall scope of the project, the survey served to help us assess the economic impact of electronic texts on humanities research and teaching, and gather important information about scholarly uses of and attitudes towards electronic texts. This report focuses specifically on the latter aspect of the survey: questions about how scholars and teachers are currently engaging with electronic texts, about their expectations and concerns, and about the standards by which they judge these resources. All of these issues are crucial in determining the rate at which electronic resources may be adopted for humanities use, and what factors may influence this rate—which in turn will have a profound effect on how electronic resources are designed and marketed. Only within this conceptual context can we expect to gain an accurate picture of the real impact, economic and structural, of electronic resources in the academy.

Survey Design and Aims

The survey was sent to 330 people, all of whom were prior users of WWP text printouts. This group was chosen not at random, but with the aim of representing a range of geographic locations, degrees of access to online technology, and professional positions—the latter largely academic, but also including a sample of librarians and independent scholars. Our response rate was better than expected; we were told by survey specialists not to hope for more than about 10%, but in fact we received 69 responses, or 21%.

Our aims for the survey were two-fold. First, we wanted to judge the practical economic impact of electronic texts on academic research, and test hypotheses about the long-term costs of using rare texts in research and teaching, as compared to the use of primary source textbases such as the WWP’s. To this end, we wanted to collect some concrete data on the kinds of costs researchers incur when using rare texts, and the sources of funding that typically support such research.

Our second, broader aim was to understand the range of attitudes and concerns that academics currently feel towards the use of electronic resources, as compared with other forms of textual material. Such knowledge, we felt, could provide important insight both into the way electronic resources are likely to be adopted in research and teaching, and into the needs that such resources must serve in order to become a useful and habitual part of scholarly life.

To elicit the more specific data on costs, we designed a set of detailed questions with multiple-choice answers to limit variation and ensure consistency among responses. For the broader questions about attitudes and text usage, however, our approach was necessarily more complex. We wanted to guide or limit the responses as little as possible, to capture the full range of respondents’ opinions. For this reason, these sections of the survey asked for open-ended comments, inviting the respondents to speak as frankly and fully as possible.

This approach to survey design has advantages and disadvantages. The advantages, as stated above, are that one does not artificially limit what can be said in response to the survey by anticipating a certain range of responses and foreclosing others. However, open-ended questions make it more difficult to derive quantifiable conclusions from the data. To overcome this problem, we created a set of codes that represented the specific themes and issues that we wanted to track in the responses. Each response could then be coded to indicate which themes were present, thus providing a way of identifying patterns and expressing conclusions more precisely.

Survey results on use of electronic resources

Use of Electronic Resources

Our first focus for the survey was to determine the extent to which humanities scholars currently use electronic texts for research and teaching, and the extent to which they plan to do so in the future. 46% of respondents described some current use of electronic texts or online resources in their research. 40% described some use of electronic resources in their teaching (including use of printouts generated from online resources). Altogether, 59% of respondents described using electronic resources in some manner in their current work as scholars and teachers.

43% of respondents described some sort of future use of electronic resources in their research. 52% of respondents described future use of electronic resources in their teaching. Altogether, 67% of respondents described some intention to use electronic resources in the future, in their teaching or research.

The teaching responses were divided fairly evenly between junior and senior faculty with a slight preponderance of the latter, but research use showed nearly twice as many senior faculty as junior faculty. These ratios were also evident in the responses about future use.

As part of the coding process, we distinguished between “basic” use of electronic resources and “advanced” use, where “basic” use involved simple on-screen reading, basic searching, and using printouts for research or teaching. “Advanced” use involved activities requiring some more sophisticated framing of research questions, or a more complex interaction between user and text. Advanced classroom activities might involve online editing by students, or computer-assisted textual analysis, or the creation of a class web site. Advanced research activities might anything from detailed searches using metadata, to the creation of an electronic edition. Of the 28 respondents who indicated that they used electronic resources in their teaching only 6 described what we would categorize as “advanced” use; the same percentage was evident in the descriptions of future classroom activities. Of the 32 research respondents only 4 described advanced use, and this percentage dropped (2 out of 30) when looking at future research use.

Besides the “basic/advanced” axis of comparison, we also noted the level of detail in which respondents described their work with electronic texts, to gauge their knowledge of electronic research and teaching possibilities (although with the caveat that absence of detail in a short-answer survey does not necessarily indicate lack of knowledge). 14 respondents described their work with electronic resources by mentioning a specific research activity which corresponded to a specific set of research questions: for instance, “search for pamphlets and relevant commentaries on primary texts”. The more frequent response was of a more general nature, at the level of “get access to rare materials”.

The range of research projects and activities was surprisingly broad. Respondents described stylistic and grammatical analysis, manipulation of texts within a corpus using metadata to assist discovery and analysis, comparative work with textual variants, and use of concordances, as well as specific searches for words, phrases, references, etc. There was also considerable interest in using finding aids and metadata to assist with the early stages of research: discovering resources, getting information on provenance, material history, and so forth.

A similar breadth was evident in the teaching projects described. The most frequently mentioned teaching use was the creation of online textbooks or print anthologies drawn from online sources, which confirms the WWP’s own experience. In addition, however, respondents mentioned having students compare different editions online, having them do online editing projects, teaching them literary research methods with online resources (both primary and secondary sources), performing structural analysis of narrative, and teaching analytical search methods. Some also mentioned developing class web sites, whether to provide access to online course materials, or as a showcase for students’ work.

Above all, however, respondents in both categories mentioned access to rare materials and information about them as the primary advantage of electronic texts, with teachers giving particular emphasis to the inclusion of these materials in course anthologies. This interest in access is discussed in more detail below.

If we compare the kinds of electronic research activities described with conventional research activities, we can see a range of diminishing correspondence which complicates efforts to compare the use of print and electronic texts; some basic activities match their conventional counterparts very well, but the more complex activities seem in many cases to be profoundly different. At the level of discovery, where the researcher is seeking textual data (either looking for relevant texts or for particular passages), the activities correspond quite closely, although the process is much easier with electronic resources. Similarly, discovery of textual patterns at an observational level (what is termed close reading in literary studies) is quite similar; with electronic assistance the user can speed up the process or render it more precise and exhaustive, but the process itself is more or less unchanged. The quantitative study of textual patterns, however, would be nearly impossible with conventional texts (although not unknown; the New Shakespeare Society of the 1870s put a great deal of effort into this kind of work). Discovery of textual patterns using the kinds of structural encoding which electronic texts make possible is again almost inconceivable with conventional texts and has no real counterpart in traditional literary study.

Uses of electronic resources in the classroom similarly display a diminishing correspondence with use of traditional materials, as complexity of work increases. In providing access to rare materials, electronic texts provide an easier alternative to photocopies, although their added readability (through control over font size, layout, and other aspects of display) is a feature which cannot be duplicated in photocopied originals. The electronic medium also provides for student collaboration, but teachers have long known how to foster collaboration without electronic assistance; the new medium simply provides a convenient infrastructure whose convenience may produce better results. Computer-assisted textual analysis, however, is likely to be unlike the work most students do with conventional texts, and the quantitative difference in the amount of data one can generate with these methods amounts to a qualitative difference in the research that is possible: instead of focusing on the process of observation and data-gathering, it allows the student to focus on the textual patterns revealed and the kinds of analysis that produce them. In short, it gives an entirely different kind of insight into the way texts work.

Scholarly Attitudes Towards Electronic Resources

The survey responses revealed a number of interests and concerns which are of particular interest because they could have a substantial effect on the adoption of electronic resources. These attitudes are also of interest for what they can tell us about users’ needs and what they expect from electronic resources.

General Attitudes Towards Electronic Texts

24 expressed generally negative attitudes towards electronic texts (not just expressing specific concerns, but indicating a fundamental dislike or distrust of electronic texts). Of these, 5 expressed a very strong negative impression.

30 respondents expressed generally positive attitudes towards electronic texts, including a desire to learn more about them, a desire to use them in the future, or a general optimism about their role in academic work. Of these, slightly over half expressed a very strong positive impression.

There is a slight overlap between these groups: 7 respondents expressed both positive and negative overall impressions. Of these, 3 expressed a strong positive impression and a mild negative impression. None expressed a strong negative impression.

22 respondents expressed neither a positive nor a negative overall impression, either because their responses were too vague or incomplete to express any emotional color, or because they had a neutral opinion of electronic texts.

Desire for access

52 respondents (75%) mentioned increased access as a strength of electronic texts.

Interest in issues of access was apparent throughout the survey responses. However, respondents used the term with a range of meanings that deserved closer attention. The domain of the word “access”, first of all, seems to split between the idea of gaining access to specific materials, and the broader notion of making materials generally easier to use. In the former category, rare texts and non-canonical texts were of paramount interest. Scholars indicated that one of their chief reasons for using an online textbase would be to increase their access to materials that are difficult to find in print form for teaching and research. The concept of access here engages with the boundaries that physical scarcity, cultural positioning, and fragility have placed around these texts—issues that affect women’s writing to a disproportionate extent. In its broader frame of reference, on the other hand, the term “access” (or “accessibility”) expresses a desire that textual materials of all sorts be made available in more useful forms. For student use, this could mean providing a modern-spelling version, or including annotations or glossaries; for research purposes, it could mean the ability to view color images of manuscripts which reveal invisible details, or the ability to compare variant editions of the text.

Access, in the survey responses, had additional connotations as well. For some, it was tied to democratizing the use of rare texts: reducing costs and removing obstacles for students and researchers of limited means, and opening up access to these materials to a much broader audience. Others raised technological issues, noting the increased importance of a networked environment for research and the growing ability to work with textual materials from home, office, or while traveling.

Finally, networked access raised the issue of the “accessibility” of digital resources in the sense of their user interface and functionality.

Concerns about accuracy and reliability

24 respondents (34%) cited concerns about the accuracy or authority of electronic texts; for about half of these, concerns amounted to a profound distrust and a potential or actual reason not to use electronic texts. All of the respondents who indicated a strongly negative overall impression of electronic texts also indicated a strong concern about their accuracy.

While the trustworthiness of the text would seem to be of self-evident importance, what was less predictable was the particular forms that concern for accuracy took in the survey results. Anxiety about the accuracy of electronic texts was so acute that some respondents discussed it even in answer to questions on other subjects, and it clearly represented the single largest obstacle to general scholarly use of electronic texts.

Having said this, however, several other points of interest should be noted. First, concern over the accuracy and reliability of electronic texts was much greater in scholars’ discussion of their research work than in their teaching. Very few respondents cited concern for inaccuracy as a reason for not using electronic texts in their classrooms, whereas a far greater number said that they would either not use electronic texts in their research, or that they would use them only where they could also check them against the originals.

Second, survey respondents expressed two quite different kinds of distrust of electronic texts. The first was an essentially pragmatic sense that currently available texts are not yet being produced to scholarly standards: an opinion based on observation and use of these materials. Respondents in this category seemed able to imagine the possibility of reliable, accurate electronic texts becoming available in the future, and to imagine using them when they did. Alongside this group, however, was another that tended to express concerns about the fundamental nature of electronic texts. Respondents in this group tended to speak of a gut-level sense of the “book-ness” of books and their aesthetic qualities (“aesthetic” here meaning the physical and cultural properties of the book form as well as its attractiveness)—these issues affecting the perceived trustworthiness of the electronic text and its cultural positioning rather than its actual transcriptional accuracy. Concerns of this nature seem more likely to persist even if electronic texts do become a standard and reliable form of scholarly resource. Scholars often attributed these reservations to the nature of their scholarly training, giving these concerns an intrinsically greater durability than the more pragmatic worries described above.

Another frequently cited issue of textual reliability was the question of editorial integrity, including both the textual choices made (choice of edition, editorial treatment) and the scholarly credentials of the editors involved. Most frequently, scholars noted that these choices are not sufficiently foregrounded in most electronic texts, leaving the reader with no sense of what sort of text he or she is using.

Interest in functionality

26 respondents (38%) cited function as an advantage of electronic texts. Of these, 10 expressed a strong enthusiasm for this aspect of electronic texts.

For many of these respondents, functionality means the ability to search the texts for words and phrases, or to search collections for particular works. This general sense of the usefulness of basic retrieval is widespread, and is supported by the growing use of the World Wide Web, in which this kind of retrieval has become quite natural and expected. In addition, respondents described more specific kinds of retrieval, including keyword searching, reference searching, and searching which relies on textual markup (for instance, searching for personal names or Biblical references).

Respondents also mentioned more advanced functions which they either use currently or anticipate using in future electronic resources. These include using electronic concordances and performing statistical vocabulary studies, working with textual variants and parallel versions, and supporting the development of other online research tools and projects (for instance, databases or hypertexts).

Concerns about the loss of the physical book

20 respondents (29%) cited concerns about electronic texts centering on the absence of the physical book. These concerns included a need to refer to physical details of printing and binding, or an aesthetic desire to touch or see the book. While to some extent (as with concerns over reliability) this more aesthetic concern can be thought of as a conceptual gap between the user and the text—evidence of a paradigm shift which may take time to become naturalized—we also need to take seriously the need for information about the physical object to support certain aspects of research. The WWP currently encodes information about the collation, pagination, lineation, and layout of the original, as well as identifying the source copy. We are also exploring the possibility of including images of title pages, illustrations, and perhaps in some cases whole texts.

Concerns about technical obstacles

13 (25%) respondents cited concerns about technical obstacles which make it difficult to use electronic resources. These obstacles included access to computers and networking, potential software incompatibilities, the difficulty of installing and maintaining CD-ROM resources, and the need for additional training.

The role of annotations and editorial intervention

Only 5 respondents mentioned a preference for having apparatus or annotations. 3 respondents mentioned a concern about the mediation or “cooking” that takes place in the preparation of an electronic text. 1 respondent indicated that the raw, unmediated character of electronic texts was an advantage.

The role of the scholar

Closely tied to the question of editorial integrity above is the role of the scholar in the preparation of electronic texts, and how that role is imagined and evaluated. Concerns about the trustworthiness of the text were quite often framed in terms of a desire for a recognized name and the authority it carries, as a way of assessing the value of a given edition. The lack of such authorities was cited by some as a crucial difference between the cultures of electronic and print media.

Even more prominent in the survey responses, though, was an engagement with the issue of scholarly intervention and its role in the creation of textual resources. A few respondents said that they valued the electronic medium particularly where it offered easy access to an unedited, unmediated version of the text from which they could draw their own conclusions. However, a number also said, on the other hand, that they valued the input of a scholarly mind, and preferred to use texts edited by a trusted expert rather than the original source. This desire for scholarly intervention, though, was offset by a concern for the untrustworthiness of editing in many electronic texts.

These are issues that arise with equal potency in the realm of print texts, but in the context of the electronic medium they take on a peculiar force, since they affect the adoption of the medium as a space for scholarly research. The role of the scholar is particularly important because as yet few scholars are actually involved in the creation of electronic texts. This is both the cause and the effect of the current institutional positioning of work in the new medium: professional assessment does not yet give much (if any) credit for the preparation of electronic resources, and scholarly publication still has a sense of awkwardness about the citation of electronic materials. As a result they remain peripheral to the mainstream of academic work.

Finally, there was some concern—though not as much as might be expected—about specific scholarly choices: about theories of editorial method, which copy-text is chosen, methods of transcription, etc. On the whole people seem willing to trust the preparers of textual resources as long as they feel them to be trustworthy. Presumably this trust could also be extended to digital resources once these become a more familiar part of the humanities landscape. Interestingly, no mention was made of the integrity of the encoding or electronic preparation of the text. From this it appears that scholars are not yet aware of the kinds of specific choices that go into the preparation of an electronic edition, and the effects these may have on its quality—both as scholarship and as a digital product.

Conclusions

At this stage, any conclusions based on the survey data are necessarily provisional. However, some general points emerge and are worth mentioning. First, it is clear that a substantial number of humanities scholars and teachers are using electronic resources, and still more plan to use them in the future. At present this use is primarily concentrated on simple activities, and is by no means central to most scholars’ work. However, the dependence of humanities research on primary sources has meant that this area has lagged behind other disciplines in adopting digital materials [1], since electronic primary source texts in the humanities have been and remain scarce. The responses to this survey indicated that even most of those who did not currently use electronic texts at all—whether through lack of materials in their area or through concerns about accuracy—would be interested in using them if they could find accurate, useful materials in their field. The problem to address thus seems to be one of supply rather than demand: if reasonably priced, reliable collections of source materials are made accessible, it seems likely that they will be used.

What impact might this use have on humanities scholarship and teaching? The economic impact will be significant, although not through a simple reduction or increase of the cost of teaching or research—rather through a redistribution of costs and a shifted emphasis in what they purchase. Clearly the cost of production of electronic resources is considerable, but the incremental cost of providing access to each additional user is extremely small. In print, by contrast, the intellectual production cost is borne invisibly in the salaries of the researchers who write books and prepare editions, but the incremental cost of providing copies to individuals is quite high; maintaining stock, keeping a book in print, cataloguing it and preserving it in a library, and so forth. [2]

Several points follow from this difference. With electronic resources, the individual user or the individual use is no longer the appropriate unit of purchase: students do not and should not buy access to electronic texts in the same way that they buy a textbook. The only real cost of distribution for electronic texts is the administrative overhead of maintaining licenses, and it is pointlessly cumbersome for the electronic distributor to sell individual licenses; purchases at the institutional level make far more sense. Pedagogically too, licensing electronic texts in bulk and at the institutional level makes good sense, since it encourages exploration of new material-and engagement with what is different and new about electronic texts-without the sense of mounting incremental costs.

If this model of institutional licensing does become the standard for distribution of electronic resources, the costs of using textual materials may be fundamentally transformed. Taking the place of the various individual costs of using printed texts—for textbook and course packets, travel for archival research, purchase of microfilms or photocopies, interlibrary loan, storage and cataloguing—we see the comparatively centralized cost of access to the electronic resource, and of maintaining the infrastructure which makes that access possible. One effect of this transformation is that questions of access to information are relocated to the institutional level. Where previously individuals made individual purchasing choices, and sought access to research materials on their own through whatever channels and with whatever funding they could find, the institution would now make purchasing decisions which affect the entire institutional community. This change may add tremendously to the leverage of those decisions, but it also may alter their politics profoundly, in ways which are very hard to predict at this point.

These reflections and conjectures remain to be filled out by further research—that of the WWP and others—but they may suggest directions that research should take in order to grasp what is at stake.

Notes

1. Pavliscak, Pamela, Seamus Ross and Charles Henry. 1997. Information Technology in Humanities Scholarship: Achievements, Prospects, and Challenges: The United States Focus. ACLS Occasional Paper, no. 37. American Council of Learned Societies. http://www.acls.org/.

2. Electronic resources therefore scale very well; once produced, they can be made widely accessible without any substantial distribution costs. This is incidentally why it is doubly important that electronic resources be designed and built for the long term, since it is only in their longevity that they can bear their high cost of production.