First thoughts on SF DORA (https://am.ascb.org/dora/)
The following was posted in The Node as a response to a call for comments on the SF Declaration of Research Assesment (https://thenode.biologists.com/san-francisco-declaration-on-research-assessment/news/)
The declaration is a very important step forward to untangle the situation we have got ourselves into. However, the size and the direction of the step will be determined by how, who and how firmly the resolutions are implemented. Some habits are difficult to quit and one can see people not mentioning the impact factor (IF) explicitly but using it implicitly valuing publications in the same journals as a proxy for â€˜qualityâ€™ and, basically, perpetuating the situation. As the Leopard says in the famous novel of its name: â€˜everything has to change for everything to remain the sameâ€ . There have been already some voices of scepticism in this direction. However, let us use the concern to guide us moving forward and take the opportunities that lie within San Franciscoâ€™s DORA.
DORA is, above all, a commitment of individuals and institutions to a common sense principle: that the value of a piece of scientific work cannot be determined by a publisher, that scientists should be the arbiters of their own work and that judgement should be done in a transparent and considerate manner. The history of Science shows many errors of judgement (the list, while interesting would be long to discuss it here) but we are not talking, for the most part, of world changing discoveries but of something simpler, the piecemeal and steady increase in knowledge and the use that this has in allowing people to pursue it. And here it is that we, as a group, have allowed publishers to use us to create a tangled web of control over our activity that stems from their power and position.
One often hears the question â€œhave you seen the latest paper of so-and-so in Cell/Nature/Science (CNS)?â€ And when you ask what is it about, after a few mumbles we are told that IT IS IN Cell/Nature/Science; as if this were the seal of authority for the validity of the publication, whatever its content. It is like those experiments in which the colour of the water determines its taste. DORA suggests that this should end and that we should begin to bother to read and consider the works on their own merits: water is water, whether pink or blue. Like many of you I have often heard that when one is faced by a large number of applicants for a position or a fellowship, the â€˜publication recordâ€™ (and we know what this means) is a quick way (the best?) to sort people out. I am not going to deny that very good science is published in High IF (HIF) journals, but I also know that this is not the best measure of Science. I have suggested before and do here again, that a good practice should be to ask applicants to list their three or four best publications and write a paragraph explaining their impact and potential. Not so much the selection but its justification will tell you a great deal about the applicant and put the rest of the application into perspective. It is, I believe, suggestions like this which in the spirit of DORA, will change the attitude of individuals and allow us to move forward to the values of the past (that rest firmly on the insight and projection of a piece of research) but facing the challenges of the future (increased technical and data gathering abilities and growth of science based population).
As for the journals, DORA gives them responsibilities and we shall see if they rise to them. Here I have less faith. Nature magazine has not signed DORA and has explained why (https://bit.ly/17AFlYF)Â . It has also been pointed out that Nature (and others) have been encouraging care and thought with the IF for years); this is well known and welcome. Personally I take their refusal to sign as a statement of honesty: they recognize that agreeing to DORA should have some consequences for the scientific publishing world and they are not prepared to meet them (they express it in a different manner but this is essentially it). After all, Nature is a commercial enterprise and what they have to do is maximize profit and market value, both of which depend on their IF and DORA, implicitly, bears the potential to undermine these (they have recently claimed that it costs them around Â£30K to publish a paper). So, why should they adhere to it? They have made it clear that the IF is something created by the community which benefits them/us so, quite rightly say that it is not their doing and wonder what gain is there for them in ignoring it. They point out that many of the statements in DORA have a very broad sweep, and they might be right but this is good. Let me put forward an example of something implicit in DORA for publication policies.
I have heard editors often say that the reason to keep rejection rates high is to try to boost up the IF of their journals (there is no point in denying this; it is a fact). If the IF is not any more a measure of the merit of a publication, there is no reason to keep rejection rates as high. I do hear people saying that this is not true, that what they want with the rejection rate is quality and that they will keep their rejection ratesâ€¦â€¦â€¦..if this happens, and in the short term IT WILL happen, we shall be subject to IF by stealth. But still, let us continue with the argument. PLoS, as I have discussed before (https://bit.ly/Zi7OKH) is a very good example of the failure of this policy. PLoS Biology was born with many ideals in mind, but one of them was to create an OA journal that would compete with NCS. The OA bit has succeeded (and we are all grateful for that), the competition with NCS has not and this despite their very high rejection rates. For example, compare articles in April 2013 between them and another high profile PLoS journal: PLOSBiology 19 publications (IF 11.45) PLOSGenetics 69 publications (IF 8.69). They can talk to me about quality but actually it is not justifiable that two sibling e-journals have such disparate publication rates. The data also show the poor relationship between IF and rejection rate. Why the difference?. Because they (and many others) believe that increasing the rejection rates will increase the IF; it does not work like that. Butâ€¦â€¦.. one of the consequences of DORA should be that, since they do not have to pursue the HIF value, one would expect a rise on the publication rate of some of these journals. A good reason for them would be that they might want to capture the papers that will make a real impact (which might not be the ones that are presumably doing this at the momentâ€¦â€¦). Let me repeat it, not doing this will be IF by stealth. Â NB: I am not talking about â€˜metricsâ€™ here, I am talking about IF and how it is understood, evaluated and perceived.
Along these lines, let me spell out what I see as a most important consequence of this declaration and how it can be harnessed for an all around win. If we waive the value of the IF in the assessment of careers and science, we get back our rightful position of running the publication of our work and have the journals compete for our toilings rather than the way it is now (the other way around, actually): we compete amidst each other for the space in the journals. Think.
What is the incentive of sending your next piece of work to a journal with a high IF? Let me become more clear in the context of the interests of The Node. When you have a good piece of work, why would you send it to one of the Cell Press journals, Nature, Science or even PLoS Biology where you know ahead of time that, if sent to review, it will linger through several rounds of review for any time between 8 months and a year, without a guarantee that it will be in the end published and, in the case of Cell Press, with the threat that it will be rejected if a related piece of work is published in the meantime? In the process, in these journals the professional editors will meddle with YOUR science, will decide WHAT experiments YOU NEED TO DO, this will make you spend time and money improving a paper 10-20% and all within the cloak of anonymity of a reviewing process which everybody admits is seriously flawed. Why donâ€™t you send your work to Development, Dev Biology or, in terms of generality, EMBO J or maybe even eLife? In these journals, though there are similarities in the general process, you will have a fair viewing, your work will be dealt with by people in your field and you will have it out soon with a minimum of hassle. If the IF is not any more an issue, you should be confident of your Science and make sure that it is out (after a fair, transparent and helpful peer review) as fast and efficiently as possible so that it can be judged by your peers. So, DORA offers an opportunity for you and also for journals like Development to catch up and regain some territory over others which have been sailing on the wave of the HIF. By increasing their acceptance rate (and this does not mean to decrease the quality of the publications) Development, Dev Biol, EMBO J can capture a lot of very good pieces of work. This will entail changes in their editorial policies, but perhaps DORA means change for everybody and we should face this together. The real question here is how confident are journals on their possibilities. I have always wondered why Development is happy being a journal that trails on the wake of, say, Dev Cell or as they say a â€˜community journalâ€. If they want to be the latter they should make sure that they rally the community with them. I think DORA is presenting them, and others, with a tremendous possibility to lead. I suspect that Nature magazine knows this and we should respect that they do not want to play ball. In making this statement, they are making their colours clear.
This is the time for us to choose between, for example in developmental biology, Development/Dev Biol and Dev Cell. The CoB should be aware of this and play their cards with imagination and courage.
For us, the question is: shall we seize this opportunity? As Leslie Vosshall said (and I cannot repeat this enough because it is SO right and to the heart of the problem): â€œscientific publishing is an enterprise handled by scientists for scientists, which can be fixed by scientistsâ€ (https://bit.ly/Q52zvY). DORA is an opportunity and the question is: are we bold enough?
In response to a comment on the relationship between rejection rates and quality
Let us not be naÃ¯ve on the issue of rejection rates and scientific quality.
There is 20% rejection rate and 80% rejection rate. The first is clearly likely to lower the quality of the science that a journal publishes, the second one is more interesting. Imagine an aspiring â€˜glamour journalâ€™ (we are not allowed to talk about IF so let us not to) with a 80% rejection rate, publishing very good science (it is possible with this %) but, for reasons that we might not get into (become more elitist, increase its glamour factor, etcâ€¦.) might want to increase the rejections to 90% (this happens, believe me). Given the current trends, it is likely that the 20% of the papers which could be published are very good and I can assure you that what paper falls in the 10% that would be rejected (in the change of policy) has nothing to do with quality but rather with perceived impact or, simply, arbitrary decisions. So, in this case keeping the rate at 80% would not lower the quality of what is publishing and, frankly, would not change the glamour factor that much.
In this context, the comparison between PLoS Genetics and PLoS Biology to which I have often alluded to (see above comment) is a good example of this. Do you think the papers in PLoS Genetics of lesser scientific quality than those in PLoS Biology? I donâ€™t. Do you really think that papers in Dev Cell are better than those in Development? I donâ€™t. In fact what I often find is that â€œa DevCell paperâ€ is a hyped version of a Development paper and many papers which end up in Development come from journals like Dev Cell where, after (some times a few rounds of) review they are rejected becauseâ€¦â€¦reviewer 3 or simply, the editor was not â€˜convincedâ€™. Â And the Development papers tend to be good. All this refers to rejections after review, not editorial rejections for which I suggest people to read a recent article Â (https://www.sciencedirect.com/science/article/pii/S0169534713001262) and remember that what it says is not restricted to Ecology and Evolution. Editorial rejections (which cannot be on the basis of the science because they do not involve peer review, though in some journals do involve an interaction with the editorial board) are, in general, the most common ones.
So, in the age of e-publishing quality should indeed be the criterion but it seems to me that it is not possible to justify publication rates below 10%, which is what glamour journals often do
Personally I am fine with this, as long as it is made clear that the decision is a matter of opinion and is not justified on scientific grounds. I have never understood why in the occasions of two glowing reviews and one negative, the negative wins.
DORA opens the door to many new possibilities but for them to be realized we are going to have to leave behind many hang-ups.