Improving Journal Ranking: Alternatives to Gross’s Citation Counting Method

When a librarian decides which journals to subscribe to, or when a researcher judges where to publish, both rely on the idea that some journals matter more than others. But how do you measure that importance objectively? The answer that has shaped library science for nearly a century is citation analysis – counting how often a journal is cited by others. The method began with a simple count proposed by Gross and Gross in 1927, yet that simplicity hid serious flaws. This post explains why straight citation counting can mislead us, how the Indian scientometrician I. N. Sengupta devised a clever correction, and what we learn by comparing different ranking methods side by side.

The origin of citation counting 🔗

The story starts in 1927, when P. L. K. Gross and E. M. Gross published a study in the journal Science titled “College libraries and chemical education.” They examined every reference in one year’s issues of the Journal of the American Chemical Society and tallied which periodicals were cited most often. Their goal was practical: to help small college libraries decide which chemistry journals were essential when budgets were tight.

This was the first citation analysis ever conducted, and the principle behind it became foundational. Gross and Gross treated the raw count of citations a journal received as a direct measure of its importance. More citations meant a more valuable journal. This idea seeded everything that followed in bibliometrics, including Eugene Garfield’s later work on citation indexing and the impact factor.

How the method works 🔗

The Gross method is appealingly straightforward. You select a set of source journals in a field, you go through their reference lists, and you count how many times each cited journal appears. Rank the journals from most-cited to least-cited, and you have a list of the “core” periodicals for that subject. Gross and Gross even noticed a pattern that later matched Zipf’s Law – a small number of journals were cited very frequently, while a large number were cited only once or twice.

For libraries with limited funds, this was gold. It gave an objective, replicable, quantitative basis for collection development decisions, replacing guesswork and personal preference. That objectivity remains the strongest argument in favour of citation analysis even today.

Problems with country and language 🔗

The trouble is that a raw citation count measures more than just quality. It measures visibility, accessibility, and the citing habits of researchers – and these are heavily skewed.

Consider language bias. Papers published in English consistently receive more citations than equally good papers in other languages, simply because English is more widely read. A study of the Science Citation Index found that English-language articles yield far higher citation impacts than non-English ones, and that citation databases provide uneven coverage of foreign-language journals to begin with. For a country like India, where some valuable research appears in regional or specialised outlets, this means genuinely important work can look unimportant in a raw count.

Then there is country bias. Researchers tend to cite work from their own nation more readily, partly because they read national journals and partly because of professional networks. Studies have shown that scientists are more likely to cite papers published in national languages and that authors from large, well-funded research communities receive more citations than those from smaller ones. The result is that the citation totals of journals from the United States and Western Europe are inflated relative to journals from elsewhere, regardless of the actual research merit involved.

A raw count also captures self-citation and the sheer size of a journal. A large journal that publishes hundreds of papers a year will naturally collect more citations than a small specialist journal, even if the small one is more influential per article. Garfield’s later insight – to divide citations by the number of citable items – was a response to exactly this size problem.

Sengupta’s weightage formula 🔗

One bias that the Gross method completely ignores is chronology. This is the problem that the Indian scientometrician I. N. Sengupta set out to solve, and his correction is a notable contribution to the field from India.

The post-war journal problem 🔗

Sengupta observed that ranking lists are built by counting citations with equal weightage given to every citation, regardless of when the cited journal began publishing. This sounds fair, but it quietly punishes newer journals. As he explained in his work, the position occupied by post-war journals may not be an accurate index of their real demand or usefulness.

Why? Because a journal that started in, say, the 1960s has had far fewer years to accumulate citations than a journal founded before the Second World War. Many post-war journals had been in existence for less than twenty years when these ranking studies were done, so their citation counts covered a much shorter window than those of older, pre-war journals. A young journal of excellent quality simply has not had time to build up a citation record comparable to an established one. Raw counting therefore discriminates against new journals through no fault of their content.

How the formula corrects the bias 🔗

To level the playing field, Sengupta proposed an off-setting weightage formula. Instead of treating all citations equally, the formula applies a corrective weight that accounts for the shorter period during which newer journals could have been cited. In effect, it boosts the credit given to citations of younger journals so that their scores reflect performance per unit of available time rather than raw accumulation.

This adjustment changes the rankings meaningfully. When the weightage is applied, several post-war journals rise to positions that better reflect their current usefulness to working researchers, while the artificial advantage of older journals is reduced. The correction does not throw out citation counting; it refines it so that the age of a journal stops distorting the result.

Application to microbiology and biochemistry 🔗

Sengupta did not leave the formula as theory. He applied it to real disciplines. In one study he used the weightage formula to rerank periodicals in the field of microbiology, producing a revised list that he recommended in preference to an earlier ranking he himself had compiled without the correction. He carried out similar work on the literature of biochemistry, examining how the growth of the literature changed the ranking of periodicals over time.

Sengupta was a major figure in Indian library and information science more broadly. He is also widely credited with helping define the scope of bibliometrics and related fields like scientometrics and informetrics, which makes his methodological work all the more significant for students in this country.

Comparing citation methods 🔗

Sengupta’s formula is one of several attempts to improve on raw counting. Comparing these methods reveals an important lesson: no single number tells the whole story.

Different methods, different insights 🔗

Garfield’s impact factor, introduced in the 1960s, normalises citations by dividing them by the number of citable articles, which addresses the size problem that Gross ignored. Later indicators went further. Metrics such as SNIP (Source Normalised Impact per Paper), SJR (SCImago Journal Rank), the Eigenfactor, and the h-index each correct for a different weakness – field differences, the prestige of the citing journal, or the longevity bias in older papers.

What is striking is that these indicators do not agree with one another. A detailed review of business and management journals found that although the various metrics appear highly correlated, in practice they lead to large differences in journal rankings. A journal can rank near the top by one measure and slip considerably by another. Each method, in other words, illuminates a different facet of what “importance” means.

Beyond simple counting 🔗

This is why modern bibliometrics treats citation counts as evidence rather than verdicts. A raw count from the Gross method tells you about gross visibility. Sengupta’s weighted count tells you about usefulness adjusted for age. The impact factor tells you about average citations per paper. A wise librarian or researcher reads several of these together, alongside qualitative judgement about the field.

The deeper point connects back to the biases we began with. Citation metrics do not capture the reasons a work is cited – whether the citation is praise, criticism, or routine acknowledgement. They should never be the sole criterion for assessing research merit, and they must always be interpreted in light of the discipline’s publication practices. Sengupta’s contribution matters precisely because it reminds us that even an objective-looking number carries hidden assumptions worth questioning.

What do you think? If a brilliant new journal and an old established one received the same number of citations, which one would you consider more important, and why? And in the Indian context, how should we account for valuable research that appears in regional or non-English journals when we rank scholarly work?

References

1. https://www.science.org/doi/10.1126/science.66.1713.385
2. https://www.tandfonline.com/doi/full/10.1080/0194262X.2023.2238013
3. https://scholarworks.calstate.edu/downloads/6w924c50c
4. https://arxiv.org/pdf/astro-ph/0401228
5. https://pmc.ncbi.nlm.nih.gov/articles/PMC3826058/
6. https://link.springer.com/article/10.1007/BF02026414
7. https://link.springer.com/article/10.1007/BF00353144
8. https://www.frontiersin.org/journals/research-metrics-and-analytics/articles/10.3389/frma.2021.742311/full
9. https://arxiv.org/pdf/1604.06685
10. https://www.frontiersin.org/articles/202382