How to Evaluate Journal Quality and Spot Predatory Journals

Two different questions get mixed up under "journal quality". The first is whether a journal is legitimate: does it do what it says, including peer review? The second is how a legitimate journal ranks against others in its field. The first question usually has a yes or no answer that a few minutes of checking will give you. The second is a matter of degree, and it says less about any single article than students often assume.

What "predatory" means

The definition most people cite came out of a three-round Delphi process described by Cukier et al. (2020). Their international expert group included funders, academics and representatives of academic institutions, librarians and information scientists, policy makers, journal editors, publishers, researchers and patient partners. The final round was held in person at a Predatory Summit in Ottawa on 19 and 20 April 2019, attended by 43 participants, and produced consensus on 18 of 33 candidate items for a definition. The definition itself was published in Nature (Grudniewicz et al., 2019):

Predatory journals and publishers are entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices.

(The Nature page does not serve its text to an automated request, so we checked the wording against the peer-reviewed literature that quotes it: seventeen open-access papers indexed in Europe PMC reproduce the sentence exactly as above and attribute it to Grudniewicz et al.)

The definition is about behaviour. It does not mention open access, fees, country or being new. A small society journal from a lower-income country that is not in Scopus is not predatory. A slick website in perfect English can be.

Related problems you may meet:

• Hijacked journals. A fraudulent website clones the name, ISSN and sometimes the look of a real journal and collects fees. Articles "published" there were never in the real journal. Retraction Watch maintains a free Hijacked Journal Checker.
• Paper-mill contamination of legitimate journals. Retraction Watch reported that in 2023 Hindawi, an open-access publisher owned by Wiley, retracted more than 8,000 articles after paper mills infiltrated its special issues. In March 2023 Clarivate said that more than 50 journals had failed its quality criteria and been de-listed from Web of Science. Being indexed is evidence of legitimacy, not a guarantee for every article.

The ten-minute legitimacy check

Work through these in order. Stop as soon as the answer is clear.

Several of these steps overlap with the Think. Check. Submit. checklist, which describes itself as "a cross-industry initiative led by representatives from ALPSP, AUP, COPE, DOAJ, ISSN, LIBER, OAPEN, OASPA, STM, and UKSG". Its questions are worth reading in full before you submit anything.

Warning signs

None of these is conclusive alone. Several together are.

• An unsolicited email inviting you to submit, often flattering, often outside your field, sometimes citing your previous paper by title.
• A very broad scope such as "all areas of science, engineering and management".
• Promised publication within days or a couple of weeks.
• Fees mentioned only after acceptance, or requested by bank transfer to an individual.
• Claims of "indexing" in services that are not selective, such as Google Scholar, Crossref or ResearchGate.
• An impact factor from a provider other than Clarivate.
• No named editor-in-chief, no postal address, or a contact address at a free email service.
• A website with many language errors, dead links or copied text.
• A title that imitates a well-known journal with one word changed.

What a failed check looks like

We do not name a journal here, because a verdict on a real title would need its own evidence and an invented title would be a fake. The pattern below is what the checks turn up when a journal fails them. Suppose you find an article that fits your review in a journal you have not heard of.

1. ISSN Portal: the ISSN printed on the website is registered to a journal with a slightly different title.
2. Indexing: the website shows Scopus and Web of Science logos, but the title is not in the Scopus source list, the Master Journal List or DOAJ.
3. Membership: the site claims COPE membership, and the publisher is not on the COPE members list.
4. Peer review and fees: review is promised within a few days, and the fee is mentioned only in an FAQ.
5. Articles: the latest issue mixes unrelated disciplines, and received and accepted dates are days apart.

False indexing claims, a false membership claim and an implausible review time match the "false or misleading information" and "lack of transparency" parts of the consensus definition. Look for the same finding in another source. If this article is the only one, treat it as unreviewed.

The opposite case matters just as much. A regional journal that is not in Scopus but is listed in DOAJ, is published by a university department, describes its review process and takes months to review is low in citation rankings and perfectly legitimate.

What the metrics mean

How to use them sensibly:

• Compare within a field only. Citation rates differ between disciplines, which is why quartiles are calculated within subject categories and why a raw impact factor from one field cannot be compared with one from another.
• Check the category. A journal may be Q1 in one subject category and Q2 or Q3 in another. Say which category you used.
• Use quartiles to prioritize, not to exclude. When a search returns 400 results, reading the Q1 and Q2 papers first is a reasonable way to find the core literature quickly. Excluding everything else by rule will remove relevant studies, newer journals, regional journals and whole disciplines that Scopus covers poorly, such as parts of the humanities and law.
• Do not judge an article by its journal. The San Francisco Declaration on Research Assessment (DORA) recommends against using journal-based metrics as a surrogate measure of the quality of individual research articles. Strong journals publish weak papers and the reverse. For the studies that matter to your argument, appraise the methods yourself. The three-pass reading method covers how.

Books, conference papers and preprints

Journal metrics do not apply to books and chapters, which are central in the humanities and much of the social sciences. For books, look at the publisher (university presses and established academic publishers use peer review), the author's standing, and reviews of the book in journals. In computer science and engineering, conference proceedings are primary venues. Rankings such as CORE give a rough guide to conference standing. Preprints have had no peer review. Cite them as preprints and check whether a published version now exists.

If you have already cited a questionable journal

1. Check whether the same finding is reported in a reliable source, and replace the citation if so.
2. If the article is unique and important, assess it as you would any unreviewed report. Is the method described well enough to evaluate? Are the data plausible? Are the conclusions proportionate?
3. If you keep it, say so in the text: "reported in a journal that is not indexed in major databases".
4. If you published in a predatory journal by mistake, talk to your supervisor or library. Withdrawal is often difficult, and institutions have seen the problem before.

For supervisors and students agreeing on a rule

A workable policy for a thesis literature review: sources should come from journals that are indexed in at least one selective database relevant to the field (Scopus, Web of Science, MEDLINE, ERIC, DOAJ), from academic book publishers, or from recognized organizations. Exceptions are allowed when justified in the text. This is easier to apply and fairer across disciplines than "Q1 and Q2 only".

References

• Clarivate. (2023, March 20). Supporting integrity of the scholarly record: Our commitment to curation and selectivity in the Web of Science. https://clarivate.com/blog/supporting-integrity-of-the-scholarly-record-our-commitment-to-curation-and-selectivity-in-the-web-of-science/
• Clarivate. The Clarivate impact factor. https://clarivate.com/academia-government/essays/impact-factor/
• Committee on Publication Ethics (COPE). Members. https://publicationethics.org/members
• Cukier, S., Lalu, M., Bryson, G. L., Cobey, K. D., Grudniewicz, A., & Moher, D. (2020). Defining predatory journals and responding to the threat they pose: a modified Delphi consensus process. BMJ Open, 10(2), e035561. https://doi.org/10.1136/bmjopen-2019-035561
• Directory of Open Access Journals (DOAJ). https://doaj.org/
• Grudniewicz, A., Moher, D., Cobey, K. D., et al. (2019). Predatory journals: no definition, no defence. Nature, 576, 210–212. https://doi.org/10.1038/d41586-019-03759-y
• Elsevier. What is CiteScore? Scopus Support Center. https://www.elsevier.support/scopus/answer/what-is-citescore
• Elsevier. What is SCImago Journal Rank (SJR)? Scopus Support Center (last updated 3 September 2024). https://www.elsevier.support/scopus/answer/what-is-scimago-journal-rank
• ISSN International Centre. ISSN Portal. https://portal.issn.org/
• National Library of Medicine. About PubMed. https://pubmed.ncbi.nlm.nih.gov/about/
• National Library of Medicine. NLM Catalog: journals referenced in the NCBI databases. https://www.ncbi.nlm.nih.gov/nlmcatalog/journals
• Retraction Watch. (2023, December 19). Hindawi reveals process for retracting more than 8,000 paper mill articles. https://retractionwatch.com/2023/12/19/hindawi-reveals-process-for-retracting-more-than-8000-paper-mill-articles/
• Retraction Watch. The Retraction Watch Hijacked Journal Checker. https://retractionwatch.com/the-retraction-watch-hijacked-journal-checker/
• San Francisco Declaration on Research Assessment (DORA). https://sfdora.org/read/
• SCImago. SCImago Journal & Country Rank. https://www.scimagojr.com/
• Silver, A. (2017, January 18). Controversial website that lists ‘predatory’ publishers shuts down. Nature. https://doi.org/10.1038/nature.2017.21328
• Think. Check. Submit. Journals checklist. https://thinkchecksubmit.org/journals/
• Clarivate. Web of Science Master Journal List. https://mjl.clarivate.com/