Nice thoughts on the current situation of research evaluation worldwide as well as in the country
09 Jul 2020 - fubar - Sreekar Guddeti
Nice thoughts on the current situation of research evaluation worldwide as well as in the country
Below is the except of the commentary by Prof. Desiraju on the pitfalls of current evaluation procedures by policy makers to judge the quality of research output written on Facebook post
Here is the text copy of my recent comment in Current Science co-authored with Bhushan Patwardhan on impact factors, h-index, and pirated journals University Grants commission COMMENTARY (25th June 2020 issue) Assessing research. The slippery slope. Bhushan Patwardhan, University Grants Commission, New Delhi Gautam R. Desiraju, Indian Institute of Science, Bengaluru With the inevitable push towards better assessment of academics, new quantitative parameters have emerged such as the h-index, which is a measure of how many times an academic, typically a scientist, is cited by others in the field, and the Impact Factor (IF) of a journal, which is roughly an equivalent measure for the publishing medium itself. These numbers are a bit like retweets and ‘likes’ in Twitter. How many people like you and how many followers do you have? Isn’t it unfair that those with more followers get more retweets and ‘likes’ for putting out more or less the same kind of material? The quantification of research output through bibliometrics has become de rigeur worldwide, often even substituting for qualitative assessments that can, supposedly become subjective and therefore whimsical. However, h-indices and IFs, while they may be precise in one way, need not always be accurate with respect to judging the importance of a researcher’s work. The University Grants Commission (UGC) and the government science departments (DST, DBT) have been rightly concerned with the proper use of these parameters, noting as it were, their application to schemes of promotions, funding and recognitions in several countries. UGC also uses a simpler quantitative measure, namely the number of publications of a scientist, as a rigorous criterion for appointment and certification as a research supervisor. Regulatory actions by the UGC and the science departments, while undertaken with the best possible intentions, have led to at least two unfortunate consequences in the Indian context. The first concerns careless application of bibliometrics while deciding appointments, promotions and awards. Research points to things that are not visible. Research is not about repeating what is obvious. Just doing incremental things, merely extending what has been said and done, does not constitute meaningful research. You may receive a publication from such work, but you can be sure it is not going to be impactful or with long-lasting influence. There is now the distressing trend of appointing and rewarding people merely because they have publications in high IF journals. A pedestrian ‘follower’ paper from India can appear in a high IF journal for various reasons, including but not limited to the patronage sometimes extended by a First World referee towards an author from the Global South, a sense of noblesse oblige as it were, or a tendency to ‘allow’ an Indian follower in the same field to publish in a high IF journal if he/she cites a big ‘leader’ scientist from the First World. Such a ‘leader’ may well be the referee; the Indian paper gets published but it is never cited. Such ‘follower’ papers from the Global South will not be cited where it really matters. Indian committees for appointments, assessments or awards at the central government level often go, sadly enough, by IFs of the publication journals of candidates. These committees are necessarily of a general composition. They cannot be expected to go into the finer but more crucial details of the candidates’ research. And here’s the rub: the devil is in these details and it is only this devil that can discriminate between truly insightful and adequately competent work. It’s not a surprise therefore that average academics get elevated to positions of authority in India because it is normally only the receipt of such awards that elevates one to such positions. This anomaly concerns researchers at the higher end of the research spectrum. Let us now consider the second distressing consequence of using bibliometrics, but this time at the lower end. Policy makers and administrators worldwide have been concerned for some time that research is being paid for twice over: the first time when it is funded and the second time when journal subscriptions are paid. Scientists should not be charged twice: once to undertake research and then to view the outcomes of that research. This has led to the appearance of a new type of journal, namely the Open Access (OA) publication. In an OA journal, an author pays a one-time fee to publish a paper. Subsequently, its access is open to anyone. So if a government funding agency earmarks a certain amount (say 15%) of a research grant towards OA fees, it would pay for research just once. The OA model has been successful and excellent OA journals now exist. The model has been widely adopted by European governments and there is little doubt that India should follow this path, because it is the future. Now for the flip side. The UGC regulatory provisions for appointment and accreditation especially in smaller colleges and universities in the country have led to the disgraceful phenomenon of predatory journals that adopt a perverted version of OA. These journals more or less publish any submitted paper without the usual protocols like screening, refereeing, revising and editing. Predatory journals are also able to fix IFs through fake citations. These dubious practices exploit the desperation of researchers who have found this loophole in the UGC regulations to attain eligibility for appointments, promotions and accreditations through a certain number of points to be accrued from publications. Bribe and be published seems to be the norm, as a consequence. Regrettably, India heads the list of countries in terms of the number of predatory journals published (64%) and the number of authors publishing in these journals (11%). According to the Nature Index ( 2014), a large percentage of research articles in India are being published in predatory journals defined as ‘entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial/publication practices, lack of transparency, and/or use of aggressive and indiscriminate solicitation practices’. This is a disgrace to not just individuals but their employers and institutions. Retraction Watch is full of papers from Indian academics, proving that the quality of research and publication is dubious, at best. The UGC’s Consortium for Academic Research and Ethics (CARE) promotes academic integrity and publication ethics, and aims to improve the quality of research in Indian universities. The CARE initiative to clean up research publications in India focuses on predatory publishers/journals. It has done a good job in weeding out many suspect publications. The repeated public notices, gazette notifications and circulars to institutions are sensitizing researchers to the dangers of plagiarism/self-plagiarism, publishing in predatory journals and unethical publishing practices. While all this is welcome, more needs to be done and the benchmark for research evaluation needs to be continuously raised. The UGC has to make academics and students familiar with research methodology. Publications arise from research. If the research is poor, the outputs will naturally be poor. UGC needs to be strict about evaluation of the quality of research publications, and not just compute numbers. This also means being vigilant about the quality of research supervision. How research guidance is undertaken today must be rethought even more so in the fund-starved post-COVID dispensation in which we now find ourselves. The golden mean would be a balance between quantitative and qualitative evaluation. The former could constitute a lower threshold and establish the minimum eligibility for an entry-level appointment or a preliminary/early career award. The latter should be the upper bound, directed at subsequent career advancement (promotions, higher posts), and seek to establish excellence and leadership. But implementation of both these yardsticks needs a high measure of honesty and integrity