Monday, November 22, 2010

Lessons from the El Naschie-Alexandria scandal

El Naschie Watch limits its coverage almost entirely to El Naschie and his E-infinity group followers like Otto Rössler and Ji-Huan He. But there are lessons to be learned that are much broader.


First of all the Times Higher Education (THE) ranking system used this year has humiliated THE and Thompson/Reuters, so you can bet it will never be seen again. Next year Alexandria will fall hundreds of places. THE will make sure of it.

There's a fundamental problem with Impact Factor (IF) itself, which isn't used only for university ranking schemes, but as a criterion for promotion and compensation of academics all over the world, by university administrators. It has been noted that THE didn't account in its weightings for differences between disciplines in how many citations per paper are typical. That is true but it misses the forest for one stunted tree.

El Naschie got caught because his self-citation misconduct was so extreme. Many others were less blatant and have not been called out. One proposed fix is to drop self-citations from the IF counts. That would push Alexandria out of the top 200 certainly, and from THE's perspective solve the problem -- their embarrassment. But it would do nothing about systematic mutual back-scratching between academics, between universities, or between journals as with the Ji-Huan He/Mohamed El Naschie mutual admiration society.

Ji-Huan He didn't get his ludicrous "hottest research" accolades only by self-citation, but by having his students cite long lists of his birdcage liners in papers of their own.

A suggestion has been made to implement an increasing weighting factor for citations, the farther they are from the cited paper in terms of graph distance between citer and cited, with an adjacency metric based on coauthorship, akin to Erdős number. This Rube Goldberg idea would still be gameable by the El Naschies and Hes of the world. They'd love the challenge.

These general observations about IF shortcomings are not my own. Commenters on this blog and University Ranking Watch know all about these issues. They were... uh... soon identified by various newspapers. I'm just mentioning them because of the strong feeling from readers who have commented here, and corresponded with me, that something needs to be said and something needs to be done.

Good luck with that.

There's no technical solution. No system of rules or weights will fix the problem. One cannot do better than public humiliation of worst offenders -- a non-technical solution that THE will not adopt.

We return now to our usual coverage: All El Naschie All The Time.

Translate English to Arabic
محمد النشائى El Naschie Watch محمد النشائي El Naschie News محمد النشائى محمد النشائي All El Naschie All The Time محمد النشائى
StumbleUpon.com

9 comments:

  1. These are great thoughts, Jason.

    Why don't you try to summarize all these ideas, benefiting from your experience with all the E-infinity,THE, etc... scandals, and write a (not very militant) paper criticizing the ever increasing dependence of the scientific institutes on impact factor ?

    I for one would love to see some serious discussion about the matter away from the blogoshpere. (no intention to discredit the bloggers though). After all, if the NYtimes was ashamed from citing your blog, other newspapers won't be ashamed from citing a paper published in a respected journal.

    ReplyDelete
  2. > A suggestion has been made to implement an increasing weighting factor for citations ... with an adjacency metric based on coauthorship

    Another suggestion is a page-rank like methodology:
    http://www.eigenfactor.org/whyeigenfactor.htm
    where a journal's "importance" is weighted (recursively) according to the "importance" of the journals with articles that cite it. In that methodology, the He/ElN mutual admiration society would never get off the ground, with few external citations from the real world to inflate their balloon.

    > and write a (not very militant) paper criticizing

    You are forgetting this is a very different instantiation of the "JA Rush" who co-authored serious articles in the late 80s and early 90's on lattice-packing of superballs and polytopes in higher dimensions (with the likes of Sloane, Odlyzko, and Elkes).

    > I for one would love to see some serious discussion about the matter

    Search google scholar for "problems with citation analysis" or "problems with impact factor" for decades of articles:
    Problems of citation analysis
    A compendium of issues for citation analysis
    The use and misuse of citation analysis in research evaluation
    Is citation analysis a legitimate evaluation tool?
    Why the impact factor of journals should not be used for evaluating research
    Advantages and limitations in the use of impact factor measures for the assessment of research performance
    The impact factor: time for change
    Impact factor: a valid measure of journal quality?
    Understanding the limitations of the journal impact factor
    Commentary: The power of the unrelenting impact factor—Is it a force for good or harm?
    ...

    ReplyDelete
  3. Thank you, Hossam. You are among those whose views informed mine. But my outlook is too bleak to write such a paper. I have no solution to offer and no hope that one can be found. Administrators and librarians will go on using IF because it's easy and it's there, like a candy dish at their elbow.

    You know the issues and could write a paper. The IF is understood by numerate academics everywhere to be flawed and gameable, so it would probably pass muster with the referees, if it were submitted to a suitable journal. Richard Holmes might have useful advice about such a project. As the search terms Anonymous suggested show, there is scholarship in this area.

    ReplyDelete
  4. Anonymous, I insist this blog be about El Naschie, and not about me; but I vouchsafe that no instantiation of me would have written a paper about citation schemes.

    ReplyDelete
  5. "You are forgetting this is a very different instantiation of the "JA Rush" who co-authored serious articles in the late 80s and early 90's on lattice-packing of superballs and polytopes in higher dimensions (with the likes of Sloane, Odlyzko, and Elkes)."

    So what? If Arnold deals with that, Jason could do as well, e.g., by submitting an article on this topic to arxiv.

    ReplyDelete
  6. I must admit that I was unaware of the amount of literature about this matter. I came across two papers or so in the past about IF but I didn't realize that it was that much investigated.

    ReplyDelete
  7. Please, guys, enough about me. It's not about me. I could write a paper about it but many others who read this blog could as well. I just think it's futile. Someone more sanguine would be a better choice.

    ReplyDelete
  8. I agree 100% with the argument that bibliometry is terminally broken already. The articles on He at the Thomson Reuters web sites are brilliantly embarrassing for them and the whole enterprise of bibliometry. TR probably can (and will) make the worst abuses disappear from view by removing self-citations, but, as pointed out, that won't stop organised abuse from continuing.

    What I've found nowhere is an analysis of Ji-Huan He's actual work. Is it entirely worthless or has he done something useful? How about his most cited article (bottom of this link), for instance?

    http://sciencewatch.com/inter/aut/2008/08-apr/08aprHe/

    ReplyDelete
  9. A recent post by Holmes with the emphasis on the Great Man:

    Self Citation

    ReplyDelete