IFLANET home - International Federation of Library Associations and InstitutionsAnnual ConferenceSearchContacts

61st IFLA General Conference - Conference Proceedings - August 20-25, 1995

Literature Searches In Medicine: A Comparative Evaluation of Manual and CD-ROM Search Modes

Nazli Alkan, Faculty of Letters, Ankara University, Ankara, Turkey


ABSTRACT

In the present study, manual and CD-ROM search modes are compared in terms of quality and time, based on the medical searches conducted from Index Medicus and Medline CD-ROM (Ebsco). The more effec tive search mode is determined with respect to certain quality criteria (Relevance, Coverage, Novelty). The faster mode in terms of response time is revealed, concerning the rate of the speed. The medical doctors of Hacettepe University Medical Center (Ankara, Turkey), who refer to the Center Library in 1991, are represented by 50 samples. Each of the 50 request topics is searched by the re searcher, both manually and by computer. Equivalent periods of coverage are used (1988-1991). Precision and Recall, being the measures of relevancy, and Novelty ratios are found for both modes. The source, representing the current medical literature faster, is determined as well. Coverages of both search sources, for an equivalent period, are compared. The faster mode is found by meas uring the response times. Results indicate that the CD-ROM mode is more effective than the manual one, in terms of relevance, coverage, novelty and response time. However, manual searches have r evealed higher Precision and the time devoted to manual search strategies is shorter. Since the two modes have complementary features, co-existence of both is suggested.


PAPER

INTRODUCTION

Let it be admitted that there is always a need for high quality medical search results in every country in the world, because these results have significant effects on human health. A weighty port ion of literature searches performed by several institutions in this country are on medicine. The aim of the present study is to determine the more effective medical search mode in an objective ma nner, since there is a strong subjective tendency towards computerized literature searches in Turkey. CD-ROM mode, in particular, attract the attention of most of the librarians an d users. Manual searches seem to be entirely abandoned.

In the present study, manual and CD-ROM search modes are compared in terms of quality and time, based on the medical searches conducted from Index Medicus (IM) and Ebsco Medline CD-ROM respectiv ely. The more effective search mode is determined with respect to certain quality criteria such as relevance, coverage and novelty. The faster mode in terms of response time is revea led, concerning the rate of the speed.

METHODOLOGY

The medical doctors of Hacettepe University Medical Center (Ankara, Turkey), who refer to the Center library in 1991 (821) are represented by 50 samples (n=50). Following the search interviews, an effort was spent to arrange search strategies equally. As a principle MeSH headings were selected for both modes. However, in preparing CD-ROM search strategies natural langu age terms were used, where needed, in combination with MeSH headings. In particular, when a specific MeSH heading did not exist at all, natural language terms were used. Each of the 50 request t opics was searched both manually and by computer, in the library, by the researcher. Equivalent periods of coverage were used (1988-1991). As far as 1991 searches were concerned, equivalent mo nths of the year were determined. However, there was a 6-month delay in receiving 1991 IM issues while the CD-ROM disks were usually available within the existing month. The problem was solved by installing the back month's disk each time, instead of waiting for the coming IM issue.

Each CD-ROM search output was compared with the equivalent manual search result list to find out the unique and common citations. As a result, citations were categorized in 3 groups:

  1. Unique citations retrieved from the manual search (I)
  2. Unique citations retrieved from the CD-ROM search (II)
  3. Common/matched citations retrieved from both search modes (III)
The requesters were asked for making relevance judgements regarding their search topics. Two relevance assessments, relevant and nonrelevant, were used, simply just to avoid confusions and uncerta inties of multi-category relevance assessments.

Single literature lists, categorized in 3 groups, were submitted to requesters for relevance judgements. The requesters were not informed about which group indicated which search mode. The advanta ge of producing a single list is to avoid the risk of judging the same citation several times.

Abstracts were provided for each citation to ensure healthy judgements. Original article abstract photocopies were attached to the citations uniquely retrieved from the manual searches (I). Since C D-ROM produced the abstracted outputs, there was no problem for the groups II and III.

A questionnaire was attached to each abstracted list. This questionnaire was prepared for additional relevance and novelty assessments.

Each judged literature list was reassessed on its return, to estimate Precision (P)-Recall (R) pairs for two modes. P and R, being the measures of relevance, are the two well-known performance cri teria. P is defined as the proportion of retrieved citations that are relevant, while R is the proportion of relevant citations that are retrieved. These two ratios are defined as:

P: Number of relevant citations retrieved/Total number of citations retrieved X 100,
R: Number of relevant citations retrieved/Total number of relevant citations in the collection X 100.

It was necessary to obtain the number of citations retrieved (for P), number of relevant citations retrieved (for P and R) and total number of relevant citations in the collection (for R).

In estimating the R ratio it was necessary to get the total number of relevant citations in the collection. This is completely impractical in any system of a normal size. As an alternative, "Rela tive Recall" (RR) is used commonly, to calculate R ratios. In the current study, RR estimations were based on the relevant citations retrieved. The total number of relevant citations in the collection was accepted as the sum of the relevant citations retrieved (from the groups I, II and III). Thus the R ratios for search modes estimated as follows: Following the calculation of individual P-R pairs, average P and R ratios were estimated for both modes (50 searches).

Coverage, being another performance measure, can be described in terms of how much coverage of the literature on a specific subject is provided by a particular data base. IM and Medline CD-ROM we re not evaluated in this particular sense. But the coverage of these two sources of 1991 were compared. Coverage evaluation was performed in two stages:

  1. The lists of journals indexed by IM 1 and Medline CD-ROM 2 (1991), were compared with the medical journal list prepared by Brandon and Hill 3 successively.
  2. The lists of journals indexed by IM and Medline CD-ROM (1991), were compared with each other in details.
Novelty Ratio (NR), being one of the important quality criteria in evaluating the literature searches, can be described as, the proportion of relevant items retrieved in a search that are new to the requester. NR is defined as:
Number of new relevant citations retrieved/number of relevant citations retrieved X 100.

Average NR s for both modes were estimated in our study by means of the assessments of the questionnaire and the requester judged lists.

Another comparative evaluation study regarding novelty was performed as well, to find out which of the sources included the current citations in its coverage earlier. This study was based on th e common citations retrieved (III).

Response Time (RT), being the basic time measure, consists of the time devoted to strategy preparation and actual search time. The faster mode was found by measuring the RT s. Mean RT s were est imated for both modes.

RESULTS AND DISCUSSION

Relevance:
The average P ratios of searches are as follows: Manual 84.2%, CD-ROM 75.1%. The average R ratios of searches revealed as: Manual 53.1%, CD-ROM 84%. The manual searches yielded a higher P rati o while CD-ROM searches revealed a higher R ratio.

The average P ratio of CD-ROM searches is lower than the manual ones. One of the reasons of this result might be the usage of natural language terms in the CD-ROM strategies. Some disadvantages o f such terms result in retrieving those citations which are nonrelevant. The success of manual searches in P ratios is due to some features of human brain including deciding, understanding, conce iving, perceiving, avoiding, coordinating. Besides, manual search strategies based on MeSH terms eliminate the disadvantages of natural language terms.

On the other hand, manual searches yielded a lower R ratio. One of the reasons of this result might be the advantage of natural language terms. Besides, if no specific MeSH headings are a vailable, the terms related to the topic, are searched under some broader MeSH headings; in such cases some significant relevant citations are likely to be missed. In addition, some terms migh t be hidden under a MeSH heading that probably does not exist in the strategy.

It should be mentioned that the success of CD-ROM searches in P ratio (75.1%) is higher than the success of manual searches in R ratio (53.1%). As a conclusion, it should be emphasized that CD- ROM searches were more successful than the manual ones in terms of relevance. But it should be kept in mind that in P ratio, manual searches were more successful.

Coverage:
Results of the coverage evaluation study is as follows:

  1. 29.8% of the journals indexed by Medline CD-ROM existed in Brandon and Hill's list. On the other hand, 23.3 % of the journals indexed by IM existed in this list. CD-ROM was more successf ul in terms of coverage. However, the outcome of this study revealed the fact that none of the sources were sufficient in their coverage, in accordance with Brandon and Hill's list.
  2. IM indexed 2.960 journals, Medline CD-ROM indexed 3.435 journals. CD-ROM indexed 1.142 more journals than IM. 667 of them were uniquely indexed by CD-ROM.
It is well known that besides IM, Index to Dental Literature and International Nursing Index are among the sources which are covered by Medline. This detailed coverage study re vealed the fact that 667 uniquely indexed journals by the mentioned CD-ROM product of Medline, were not dental and nursing journals in total. Only 326 out of 667 were such journals. 341 were ma inly medical and related area sources. On the other hand, IM indexed 192 unique journals. It was surprising to see that few of them were on dentistry and nursing. As far as the year 1991 was con cerned both sources did not index an amount of medical journals in common. Medline CD-ROM was found to be more comprehensive in its coverage, since it indexed 341 non dental, non nursing journals u niquely, which is apparently more in number than the 192 journals uniquely indexed by IM.

Novelty:
Average NR results for manual was 7.47%, while it was 11% for CD-ROM. Thus, Medline CD-ROM was more successful in bringing the relevant citations to the requesters' attention for the first time.

It was also found that Medline added the current literature into its coverage earlier than IM. 68.9 % of the common citations retrieved by both modes (III), appeared in Medline CD-ROM and IM in the same year. 30.7 % of them appeared in Medline a year earlier. There was no citations appeared in IM earlier than Medline CD-ROM. Thus one more time CD-ROM was found t o be successful.

Response Time:
The present study revealed that CD-ROM searches were faster than the manual ones, as commonly accepted undoubtedly. The RT for CD-ROM searches was 50 hours 35 minutes in total. The time spent for one CD-ROM search was 1 hour 42 seconds in average. The total RT devoted to manual ones was 149 hours 10 minutes. The time spent for one manual search was 2 hours 59 minutes in average.

One of the most significant results of the present research was that the time spent for CD-ROM strategies was longer than the manual ones. The time devoted to manual strategies was 4 hours 10 minute s in total. Only 5 minutes was spent for manual search strategies in average. CD-ROM strategies took 9 hours 35 minutes in total and 11.5 minutes in average.

CONCLUSION

Results indicate that the CD-ROM mode is more effective than the manual one, in terms of relevance, coverage, novelty and response time. However, it should be kept in mind that manual searches hav e revealed higher P, IM indexes unique journals which do not exists in Medline CD-ROM (1991), most of the common citations retrieved by both modes appear in IM and Medline CD-ROM within the same year, and the time devoted to manual search strategies is shorter.

Considering the results of this study, it is concluded that the two search modes have complementary features which suggest the co-existence of both modes, where possible and needed.

1 The National Library of Medicine, List of Journals Indexed in Index Medicus (Bethesda, Md.: The National Library of Medicine, 1991)
2 A List/output of journals indexed by Ebsco Medline CD-ROM in 1991.
3 A.N. Brandon and D.R. Hill, "Selected List of Books and Journals in Allied Health," Bulletin of the Medical Library Association 80 (3 1992): 237.

REFERENCES

1. Bakker, S. "Medline on CD-ROM: A Comparison". Online Review 13 (1 1989): 39-50.

2. Bivans, M.M. "A Comparison of Manual and Machine Literature Searches." Special Libraries 65 (5-6 1974): 216-222.

3. Bluhdorn, F.L. "Integrating Print, Online and CD-ROM Technologies". Health Information: New Directions. Proceedings of the Joint Conference of the Health Libraries Sections of the Australian Library and Information Association and New Zealand Library Association, Auckland, N.Z., 12-16 Nov, 1989. Auckland: Neq Zealand Library Association, 1990. pp: 66-83.

4. Brandon, A.N. and D.R. Hill "Selected List of Books and Journals in Allied Health." Bulletin of the Medical Library Association 80 (3 1992): 223-239.

5. Cleverdon, C. W. Identification of Criteria for Evaluation of Operational Information Retrieval Systems. Cranfield: College of Aeronautics, 1964.

6. Cleverdon, C.W. "Letter to the Editor." Journal of Information Science 1 (4 1979): 237-238.

7. Crawley, J. and C. Adams. "Info Access Project: Comparing Print, CD-ROM and In-house Indexes." The Canadian Journal of Information Science 16 (1 1991): 29-41.

8. East, H. "Comparative Costs of Manual and Online Bibliographic Searching: A Review of the Literature." Journal of Information Science 2 (2 1980): 101-109.

9. Ebsco Electronic Information. Comprehensive Medline / Ebsco CD-ROM. Topsfield, Mass: Ebsco Electronic Information, 1966 - (CD-ROM).

10. Ebsco Electronic Information. Medline Ebsco CD-ROM. Reference Manual. Topsfield, Mass.: Ebsco Electronic Information, 1991.

11. Elchesen, D.R. "Cost-Effectiveness Comparison of Manual and Online Retrospective Bibliographic Searching." Journal of the American Society for Information Science 29 (2 1978): 56-66.

12. Elman, S.A. "A Cost Comparison of Manual and Online Computerized Literature Searching." Special Libraries 66 (1 1975): 12-18.

13. Flynn, T., P.A. Holohan and M.S. Magson. "Cost - effectiveness Comparison of Online and Manual Bibliographic Information Retrieval." Journal of Information Science 1 (2 1 979): 77 - 84.

14. Hane, P.J. "Reflections on Co-existence: Print, Online and CD-ROM." Database Magazine 12 (1 1989): 6-7.

15. Haramburu, F., M.H. Godin and B. Begaund. "Manual or Computer - based Literature Searches: Is One Best For Pharmacovigilance." DICP, Annals of Pharmacotherapy 25 (2 1991): 214-215.

16. Lancaster, F.W. Evaluation of the MEDLARS Demand Search Service. Bethesda, Md.: The National Library of Medicine, 1968.

17. Lancaster, F.W. If you Want to Evaluate Your Library. London: Library Association, 1988.

18. Lancaster, F.W. and E.G. Fayen. Information Retrieval Online. Los Angeles: Melville, 1975.

19. Lancaster, F.W. Information Retrieval Systems: Characteristics, Testing and Evaluation. New York: John Wiley and Sons, 1979.

20. Lancaster, F.W. The Measurement and Evaluation of Library Services. Washington D.C.: Information Resources, 1977.

21. Lancaster, F.W. Vocabulary Control for Information Retrieval Washington D.C.: Information Resources, 1972.

22. Maciuszko, K.L. "The Case for Co-existence: Hardcopy and Online Searching". Library Journal 114 (6 1989): 55-57.

23. The National Library of Medicine. Index Medicus. Bethesda, Md.: The National Library of Medicine, 1960 -

24. The National Library of Medicine. List of Journals Indexed in Index Medicus. Bethesda, Md.: The National Library of Medicine, 1991.

25. The National Library of Medicine. Medical Subject Headings: Supplement to the Index Medicus. Bethesda, Md.: The National Library of Medicine, 1960-

26. Ohta, M. "A Comparison of Some Demand Subject Searches. Machine vs. Human." Bulletin of the Medical Library Association 55 (4 1967): 408-415.

27. Olafsson, H.A. "Searching the Literature: IM or Medline?" British Journal of Hospital Medicine 37 (2 1987): 161-162.

28. Pao, M.L. Concepts of Information Retrieval. Englewood: Libraries Unlimited, Inc., 1989.

29. Rapp, B.A. "Evaluating Medline on CD-ROM: An Overview of Field Tests in Library and Clinical Settings." Online Review 14 (3 1990): 172-186.

30. Roose, T. "Measuring Reference Performance: Manual or Online." Library Journal 114 (6 1989): 68-69.

31. Rowley, J.E. and D.R. Butcher, "The Search/Information Interface Project 2: Manual and Online Searching." Journal of Information Science 15 (2 1989): 109-114.

32. Salton, G. and M.J. Mc Gill. Introduction to Modern Information Retrieval. New York: Mc Graw Hill, 1983.

33. Saracevic, T. "Relevance: A Review of and a Framework for the Thinking on the Notion in Information Science." Journal of the American Society for Information Science 26 (6 1975): 321-34 3.

34. Smith, R.G. "Before You Scrap the Old Ways, Compare Retrieval Systems. Manual vs. Online." Online 1 (2 1977): 26-27, 51-59.

35. Tenopir, C. "Decision Making by Reference Librarians: When to Choose Online Databases, CD-ROM and Print." Library Journal 113 (October 1988): 66-67.

36. Wall, C., R. Haney and J. Griffin. "Hardcopy versus Online Services: Results of a Survey." College and Research Libraries 51 (3 1990): 267-276.