Code Number: 156-125-E
Division Number: VII
Professional Group: Library and Information Science Journals
Joint Meeting with:
Meeting Number: 125
Simultaneous Interpretation: No

Journal Quality in the Asian Region: Results of a Pilot Study for the IFLA Round Table of Library and Information Science Journals

Abstract

Paper

Introduction

This investigation took a more qualitative approach to understanding journal quality based on perceptions of key stakeholders - the editors, referees, editorial board members and impartial readers. The specific target of the study was Library and Information Science (LIS) journals published in Asia. This region was chosen partly because of investigator familiarity and ready access - both are editors of journals based in Asia. It is also an area in which LIS journal publishing is well established - there are more than 200 LIS journals in China alone (Gong and Gorman 2000), and even such small areas as Singapore and Hong Kong have major journals in this field. Furthermore, many of the journals are published in less developed countries, with information infrastructures that are often fragile, under-funded and ignored by government. If the pilot study contributed to a more robust infrastructure (of which LIS journals are certainly a part), this would be a bonus.

Essentially, however, this study was meant to be a pilot project for a more broadly based international investigation of journal quality. There is a widespread view that LIS journals around the world are of somewhat uneven quality, that they do not meet a common set of standards of excellence, that there may not even be such a set of standards, that journals in developing countries in particular might benefit from better understanding of quality, and that all such journals can become more effective channels for the communication of theory and practice to the various information professions within LIS. Improved journal quality might provide a more effective channel through which ideas, theories, practices and news can be communicated between practitioners, educators, vendors and other interested parties.

However, 'improved journal quality' must take into account the reality that contributors to journals write for a variety of reasons (Gorman 1999). Most commonly, we write to disseminate new research findings or ideas. The publication of a paper establishes precedents in the formation of new knowledge, and it puts the new information in the professional domain where it can be scrutinised, criticised and either accepted or rejected. It may then contribute to further discourse. The author also makes personal gains by adding to a list of publications that can be used for tenure and promotion, for gaining professional acceptance that may lead to speaking engagements, consultancy work, perhaps even awards.

There is, then, an apparent contradiction between the intrinsic and extrinsic reasons for scholarly and professional writing, for personal benefits can result from numerous papers of indifferent quality, whereas the imperative for the discipline is the discovery of new conceptual approaches and new techniques, for which the need is for papers of the highest quality. That the scholarly communication system has survived almost unchanged for so long shows its robustness, but the inherent contradictions in the system make it vulnerable to distortion under certain circumstances. If, for example, the 'publish or perish' imperative creates such demand among hopeful authors that editors are overwhelmed with manuscripts of an indifferent quality, then there is potential for the erosion of standards. This may occur if new journals start up to cater for the unfulfilled demand from hopeful authors. There is some emerging evidence, from current research by Philip Calvert and Shi Zengshi on quality and quantity in journal publishing, that this has happened in China already.

Both extrinsic and intrinsic reasons lead to publications that might be assessed by quantitative means, but also by qualitative means. The fact that paper x is cited y times is not an indication of quality, but rather that it is cited - it is available, it is in a journal held by many libraries, the author (or publisher or editor) is particularly good at self-promotion. Behind the quantifiable factors, then, are as yet untested qualitative factors, which is what has led to the present project.

Related Literature

An intriguing project was conducted by Chressanthis and Chressanthis (1993), one an economist, the other a serials librarian. They hypothesised that journal quality was dependent upon a number of cost factors, and especially the manuscript submission fee. They found that their definition of journal quality most closely correlated with the total number of journal article pages printed in the year, journal age, the editor's institutional affiliation, the manuscript submission fee, and the total circulation of the journal. There was insignificant correlation with the presence of advertising and the journal affiliation with a professional association. The measure of quality used by the authors, however, was the total number of citations to the journal: therefore, independent variables such as the number of articles printed and the total circulation of the journal would inevitably have a high correlation with this measure of 'quality'. Citation analysis will always be biased in favour of high circulation journals, which is why even Garfield (1977) cautioned against using it on its own as a measure of journal quality.

Anderson (1997) has provided another view of journal quality, arguing that it is linked to 'excessive publication' - submitting the same paper to two or more journals. His research was mainly qualitative in that he used journal editors' opinions on a number of questions as his major source of information - the same method employed in the present investigation. Editors believed their guidelines to authors gave some protection against double publication of papers, though a sizeable number admitted that their guidelines needed review and strengthening. As a sidelight, Anderson's qualitative research suggested that editors have primary responsibility for quality, referees the next most influence, with the editorial boards a distant third.

In a lengthy, thorough and original paper on journal quality, Day and Peter (1994) used qualitative methods to ask subscribers and authors, plus some editors and editorial advisors, what they thought about journal quality. From the LIS literature they created a list of quality criteria to use in their research:

• research design
• depth
• purpose
• practical examples
• rigour
• descriptive versus analytical
• presentation
• conclusions
• relevance
• focus versus generalisation
• application
• relevant references and recency
• clarity of thought
• structure
• interest.

Reviewers added additional criteria to the authors' original lists, asking the following questions:

Does the article add to what is already known?
Is the article demonstrably related to what has previously been written?
Are the arguments employed valid in terms of the body of knowledge?
Is the article easy to read?
Do the arguments flow logically?
Does the article make a difference?
Are the conclusions strong?

From all of this a number of criteria were selected as most relevant and applicable without the need for extensive explanation as to meaning, and these criteria were seen to fall into three major categories:

• prestige (of the editor, etc.)
• properties of articles within a journal (e.g. methodological rigour)
• presentational aspects.

This pilot project focussed on the second of the larger categories, because the investigators felt this was likely to be the most crucial in determining journal quality. The other categories were also included in the broad investigation, but only the disaggregated properties of the journal articles were examined in detail.

Day and Peter's qualitative approach is echoed to an extent by Nkereuwem (1997), who used Lester's method of journal evaluation that combines input measures, decision measures and output measures into an index of journal quality for ranking journals. There is a bias in this method which favours journals with a wide market reach. No doubt some quality journals are familiar to LIS professionals because of their intrinsic quality, but familiarity can also be achieved by 'free' distribution to association members or by low subscription rates, and this hardly equates with quality. This is often the case in developing countries, where even quite 'flimsy' LIS journals are well known because of their wide distribution.

Objectives and Methods

• to identify the most commonly accepted criteria for evaluating LIS journals
• to evaluate the success of LIS journals in meeting the criteria
• to suggest critical success factors for improving the quality of LIS journals
• to assess the feasibility of an international project aimed at determining LIS journal quality and success factors

These objectives were achieved through three inter-related activities. First, other writings were reviewed in order to develop tentative criteria of journal quality. Second, the investigators interviewed selected LIS journal editors in Asia, asking for their views on the factors that contribute to journal quality. Third, selected Asian LIS journals, specifically those edited by the editors interviewed, were examined by the investigators to see how closely they match the standards expressed by the editors during the interviews. Fourth, the investigators combined the finding of steps 1-3 to produce a list of common criteria of journal quality that might be used in a fully international project.

In reviewing the LIS literature for relevant previous writing on journal quality it became clear that most authors in this field have used quantitative methods. Nevertheless, their criteria could be used in a qualitative study, so they were drawn together and collated into groups of like categories. The resulting main categories were:

• qualities of the papers
• presentation factors
• aspects of prestige
• income factors

The first three categories were derived from Day and Peter (1994), while the minor category of income factors was taken from Chressanthis and Chressanthis (1993). It became clear early in the process of gathering data from Asian editors that the qualities of the papers mattered far more than the other categories, so the investigation focused on the criteria in that category.

Twenty LIS journal editors in Australia, China, Japan, Malaysia and the Philippines were approached. Some were interviewed in person, and others were asked to write brief notes on how articles were chosen for publication, and how they as editors ensured that the selected articles matched the aims and purpose of their journals. They were also asked to provide, in keywords, the criteria they used for selection or rejection for publication of a contribution, and to rank their criteria in order if they felt it possible to do so.

The initial list of criteria concerning qualities of papers was edited as a result of input by the various editors. Gorman (1999) had suggested six criteria for the evaluation of submissions to Asian LIS journals, and it was these six that the editors agreed ultimately accepted as most relevant to the assessment of paper quality/content:

• advancement of knowledge
• new information or data
• theoretical soundness
• level of scholarship
• acceptable research design
• appropriate methodology and analysis.

These six criteria were used for the next stage of the project, which involved in-depth analysis of the sample copies supplied by the editors. Papers were chosen at random for examination, five papers from each journal being the norm. Once a paper had been read, a 'score' was given (from 0 to 10, with 10 being the highest standard) based on how well it had measured against each specific criterion. The purpose was not to assess the journals, but rather to compare the criteria, so an average score was calculated for each of the six criteria, and then they were placed in a ranked list. The two highest rated categories (new information or data, acceptable research design) scored well ahead of the others. The third category (level of scholarship) was on its own in third place, with the other three grouped together at the bottom. From this analysis, the criteria can be placed in descending order of importance as follows:

• new information or data
• acceptable research design
• level of scholarship
• theoretical soundness
• advancement of knowledge
• appropriate methodology and analysis.

Analysis of the Data Collected from Asian Journals

Many editors said that they looked for new ideas and innovative thinking in a paper. For some this was the top criterion. The rationale appears to be something like this: editors need to publish a journal that attracts subscribers, both institutional and individual. One way to do this is to include articles that seem to be relevant to the common experience of practitioners who, in LIS at least, form the majority of the potential market. Editors sometimes expressed this directly by saying they wanted their journals to be 'practical' rather than theoretical. Practitioners, however, do not want to be told how to carry out their jobs as they are at present; rather, they seek new and more efficient and effective ways to work. This is why editors value 'newness' in manuscript content above other criteria. For one editor this was closely related to his role perception. He felt he had to be a 'missionary editor' (his words), trying to persuade librarians to reflect on their roles and to think in terms broader than the next crisis.

Another highly ranked criterion was finding manuscripts that matched the scope and purpose of the journal itself. For journals with a clear niche in the market this is easy enough to understand; however, as many journals seem to accept papers on almost any topic within LIS, despite stated criteria which are more specific, it is not easy to accept that this criterion has much real value overall. Perhaps it is more a matter of matching the intended market of the journal, and this means practitioners at a certain level of education, experience, interests, and so on.

An alternative view of the importance of a paper's subject content was put by one editor when he indicated that he sought manuscripts with 'significance in the area concerned' - meaning that he wanted to publish articles that made a real contribution to the discourse in the paper's chosen area. This could be interpreted as being the paper's 'impact', a quality that is difficult to ascertain with any degree of accuracy. As noted earlier, the impact of a paper is often measured by the number of citations it attracts; a recent Asian example of the method of measuring a journal's impact by citations is Sen (1999). Other editors wanted to publish articles with more research content (one suggested College & Research Libraries as a model to follow), but most said they did not receive enough manuscripts containing sound research.

An impartial assessment by the investigators of papers in the submitted journal issues suggests that editors are doing well overall at finding papers that match their own selection criteria. The highest average score for the papers was for the criterion, new information or data, and most editors had said this was their top priority. Acceptable research design and level of scholarship also scored well. The lowest average score given to the randomly chosen articles was for appropriate methodology and analysis. This is a worrying result, for if the selection of papers is skewed towards newness at the cost of accepting contributions with a dubious methodology, then quality is quickly eroded. In fairness to the editors and their journals, it must be made plain that no obvious misinformation or dubious theory was found in the course of this investigation, and the concerns expressed here are only about what might occur.

Prestige

Chressanthis and Chressanthis (1993) claimed that the affiliation of the journal had little impact on perceptions of its quality. This may be true of Western countries, but possibly not so in Asia and developing countries in other parts of the non-Western world. It seemed that the sponsoring institution played a significant part in establishing the status of an LIS journal in China, with the three highest ranked journals being sponsored by the National Library, the Chinese Academy of Sciences and Peking University. The same was true of journals with institutional or association affiliation in the other countries of the pilot study, including Australia. It should be noted, however, that only one journal in the study was a commercial undertaking, and the sample would be very different in this respect if the project were expanded to Europe and North America. Generally this suggests that in developing countries it would be beneficial to establish LIS journals with the backing of government departments, prestigious research institutes or well-respected professional bodies. The journal's age also appeared to be significant in determining prestige, with older journals viewed as more prestigious by most editors.

Presentation

It is difficult to discuss presentational aspects of journal publication simply because it is a highly subjective category, and few editors or readers claim expertise in matters of design. It is much more likely to be the publisher who has specific knowledge in this area, so perhaps a fuller project might include publishers among the key stakeholders. Nevertheless, it is clear from the pilot study that editors and their boards feel intuitively that presentation is a factor in increasing subscriber perceptions of their journals' quality. Literally all of the examined Chinese journals were using a cover design introduced within the past two years, or were using a higher grade paper than in the past, and all Chinese editors seemed conscious of layout, font and other design issues in ways that would not have troubled them even a short time ago. Several editors, however, when questioned about their technical knowledge, admitted that they felt uneasy about commenting on matters related to physical presentation and style. As one editor said, 'I think I know what I like and what looks good, but this is a very personal matter and I could not give any objective assessment of what a journal ought to look like.' Matters of presentation, then, are very much in the eye of the beholder- not a satisfactory situation when seeking to determine quality in an objective manner.

Recommendations for Improving Journal Quality in Asia

Similarly, the editorial board needs to represent a good spread of specific knowledge in the discipline, including an awareness of overseas developments. The editors of some journals admit that board members are chosen primarily on the basis of 'cronyism'; these journals tend to fare less well in our assessment of journal quality. The message here is simple - editors need to select board members who are well qualified, and who together represent the breadth and depth desired in the journal content.

Countries whose LIS journals are primarily in a language other than English would benefit from the assistance of anglophone journals in promoting their content to the widest possible audience and to get them into international discourse. One suggestion for improving access to a wider audience is for a developing country journal to seek overseas alliances, such as exchange agreements. This would result in a paper being published in the 'local' literature to the benefit of the profession in that country, but equally the best of the local literature would also be published in a leading international journal, thus allowing its ideas and information will enter professional discourse with greater speed and certainty. This kind of exchange agreement exists in a few cases; for example, College & Research Libraries exchanges one paper per year with Journal of Library Science in China, and Online Information Review is currently exploring a similar relationship with journals in China and Thailand.

For the most part, however, established journals in North America especially are somewhat parochial in their content - an exception is Library Collections, Acquisitions and Technical Services, which is so remarkable in this respect that it has figured in a review of the journal (Nisonger 1996); British and European journals do rather better, and it would be worth developing country journals exploring the possibility of article exchange agreements with titles in these regions.

Several editors said they received manuscripts 'out of field' which were rejected almost automatically. This suggests that the solicitation process might lack focus, yet almost all journals publish guidelines or information for authors. Often, however, these seem to provide guidance only on formatting of the manuscript, preparation of the copy, how many copies to provide, footnotes style, and so on. Few mention methodological rigour, use of the literature or the required depth of analysis. Several editors agreed that their guidelines needed revision and strengthening. In addition more could be done to disseminate the guidelines in order to attract better manuscripts that are clearly within a journal's scope. Many Asian journals already use their Web sites to publicise author guidelines and recent tables of contents, and this seems to be a simple and relatively inexpensive way of telling prospective authors what the editors want to see submitted - as long as the guidelines actually address substantive matters of content. Workshops held in conjunction with professional conferences are another means used by editors to inform potential authors of what is expected in an acceptable submission.

Unfortunately, existing guidelines are only a weak defence against unscrupulous authors. Many editors said that they uncover several cases annually of papers being submitted to more than one journal. Obviously many more cases escape unnoticed. Whether this severely affects journal quality, as Anderson (1997) claims it does, was not something this pilot project was able to determine. Clearly, though, the 'publish or perish' imperative in some Asian countries leads a few desperate and unethical individuals to try such tactics, and editors need better protection than they have at the moment.

Recommendations Regarding a Wider Project

How much larger? The pilot study collected data from 20 journals; the full project should expand this to 200-300 journals from around the world. These journals might be divided geographically as follows:

12 Asian countries
10 European countries
 2 North American countries
10 Latin American and Caribbean countries
 8 Africa countries
 4 Middle Eastern countries.

A project of this magnitude requires the willing collaboration of editors, as well as some support from readers of the journals. The pilot study has shown that journal editors are surprisingly willing to co-operate with this sort of research, and there should be no difficulty attracting more volunteers for a larger project. Not must editors be willing to support the project, but the team itself needs to be fairly large in order to cope with some of the languages in which many journals are published, including Chinese, Japanese, Arabic and Russian. In addition it will be important to involve publishers and related groups (graphic designers, for example) in the project so that matters of presentation can be addressed more adequately than in the pilot. Up to 10 investigators would need to be managed by the principal investigator.

All of this in our view could be managed on a budget of about US$6000. Is it worth the investment?

References

Altmann, K.G., and Gorman, G.E. (1999) 'The Relevance of "Cited by Leading Journal" to Serials Management in Australian University Libraries'. Australian Library Journal 48, 2: 101-115

Anderson, P. (1997) '"Gatekeepers" and the Quality of the Journal Literature: Findings from a Survey of Journal Editors into the Issue of Alleged Excessive Publication in Scholarly and Scientific Journals'. Serials Review 23, 2: 45-57.

Chressanthis, G.A., and Chressanthis, J.D. (1993) 'The Relationship between Manuscript Submission Fees and Journal Quality'. Serials Librarian 24, 1: 71-86.

Day, A., and Peter, J. (1994) 'Quality Indicators in Academic Publishing'. Library Review 43, 7: 4 - 72.

Garfield, E. (1977) 'Citation Frequency as a Measure of Research Activity'. In Essays of an Information Scientist. Volume 1, 1962-1973, 406-408. Philadelphia: ISI Press.

Gong, Y., and Gorman, G.E. (2000) Library and Information Services in China (Lanham, MD: scarecrow Press).

Gorman, G.E (1999) 'Library and Information Science Journals in the Asian Context'. 65th IFLA Council and General Conference, August 20-28, 1999: Papers and Proceedings. The Hague: IFLA, 1999. Code 005-118-E. Booklet VII.

Nisonger, T.E. (1996) 'Authorship in Library Acquisitions: Practice and Theory'. Library Acquisitions: Practice and Theory 20: 395-418.

Nkereuwem, E.E. (1997) 'Accrediting Knowledge: The Ranking of Library and Information Science Journals'. Library Review 46, 2: 99 - 104.

Sen, B.K. (1999) 'Changes in Impact Factor'. Malaysian Journal of Library and Information Science 4, 2: 41-46.

About the authors