Reviewing #9--How to judge your judgment
This post is in response to a couple more reader queries. To paraphrase, the queries were:
(1) If I can't understand a paper, how do I know it's the author's fault or my lack of comprehension?
(2) How do I tell the difference between things I as a reviewer think the paper should have done, either by necessity or to meet standards (mine, the authors', or the journal's) versus general betterment of the work?
(1) Aside from the case of poor English, which was addressed in Reviewing #8, there are a couple of reasons this might happen. One is if you were sent a paper that is so far outside of your expertise you cannot possibly give it a critical review. It happens, but I do think reviewers sometimes turn down invitations too quickly on this basis. Most authors want their audience to be larger than the handful of people in their specialty, so having reviewers who aren't in the center of a particular field can be very useful to editors. But if you don't know the field at all, you were not an appropriate reviewer and it's OK to turn down the invitation.
The second reason this might happen is the paper is not written well. That's on the author. Do your best to understand the paper. Base your review on your understanding of it. If you can think of alternate interpretations of what the authors wrote, say that. When I've done reviews, I have actually used phrases like, "I think the authors mean ABC, but I'm not sure. It is possible the authors meant XYZ." Indicate how your review would change if the alternate interpretation were the case, and summarize by saying something like "It would be very helpful if the authors would rewrite this part to make it clearer what they mean."
Whatever you do, don't blame yourself for not understanding the paper.
(2) As to the second question, the short answer is, there is no difference, so I'm not sure I caught the gist of the query. If you think the paper falls short in some respect, changing that will lead to general betterment of the work.
Because I'm not entirely clear on the question, let me expand a bit. "Work" can mean either the research that led to the paper or the paper itself. I'll answer the question from both points.
If "work" refers to the paper itself (as implied by the first part of the question), the kinds of things that might be important to look for are quality of the figures, complete citations and acknowledgment of previous work, organization, and, of course, the clarity of the writing. Another important element of this evaluation is whether the authors are shingling. I will discuss shingling in another post.
If "work" refers to the underlying research, you have not only the right but, I would argue, the obligation to point out the weaknesses in the underlying science. If the methods are suspect, or if the authors over-interpret the data, or if they overlook important implications of their work, you should point that out. These are just three examples of the kinds of problems you might encounter.
A common one is that the authors have not performed enough analyses to support their conclusions. This is the toughest one: the authors may not be able to perform more analyses because they are out of funding, or for some other reason. You have to decide if the paper has value with the data that were collected. If it does, advise the authors to tone down their interpretations and to add a discussion of the possible results if they were able to do more analyses. If it does not have value with the current data, explain your reasoning in some detail.
(1) If I can't understand a paper, how do I know it's the author's fault or my lack of comprehension?
(2) How do I tell the difference between things I as a reviewer think the paper should have done, either by necessity or to meet standards (mine, the authors', or the journal's) versus general betterment of the work?
(1) Aside from the case of poor English, which was addressed in Reviewing #8, there are a couple of reasons this might happen. One is if you were sent a paper that is so far outside of your expertise you cannot possibly give it a critical review. It happens, but I do think reviewers sometimes turn down invitations too quickly on this basis. Most authors want their audience to be larger than the handful of people in their specialty, so having reviewers who aren't in the center of a particular field can be very useful to editors. But if you don't know the field at all, you were not an appropriate reviewer and it's OK to turn down the invitation.
The second reason this might happen is the paper is not written well. That's on the author. Do your best to understand the paper. Base your review on your understanding of it. If you can think of alternate interpretations of what the authors wrote, say that. When I've done reviews, I have actually used phrases like, "I think the authors mean ABC, but I'm not sure. It is possible the authors meant XYZ." Indicate how your review would change if the alternate interpretation were the case, and summarize by saying something like "It would be very helpful if the authors would rewrite this part to make it clearer what they mean."
Whatever you do, don't blame yourself for not understanding the paper.
(2) As to the second question, the short answer is, there is no difference, so I'm not sure I caught the gist of the query. If you think the paper falls short in some respect, changing that will lead to general betterment of the work.
Because I'm not entirely clear on the question, let me expand a bit. "Work" can mean either the research that led to the paper or the paper itself. I'll answer the question from both points.
If "work" refers to the paper itself (as implied by the first part of the question), the kinds of things that might be important to look for are quality of the figures, complete citations and acknowledgment of previous work, organization, and, of course, the clarity of the writing. Another important element of this evaluation is whether the authors are shingling. I will discuss shingling in another post.
If "work" refers to the underlying research, you have not only the right but, I would argue, the obligation to point out the weaknesses in the underlying science. If the methods are suspect, or if the authors over-interpret the data, or if they overlook important implications of their work, you should point that out. These are just three examples of the kinds of problems you might encounter.
A common one is that the authors have not performed enough analyses to support their conclusions. This is the toughest one: the authors may not be able to perform more analyses because they are out of funding, or for some other reason. You have to decide if the paper has value with the data that were collected. If it does, advise the authors to tone down their interpretations and to add a discussion of the possible results if they were able to do more analyses. If it does not have value with the current data, explain your reasoning in some detail.
Comments
Post a Comment