Jump to content
Sign in to follow this  
Snarf

Science Isn't Broken

Recommended Posts

 


f you follow the headlines, your confidence in science may have taken a hit lately.Peer review? More like self-review. An investigation in November uncovered a scam in which researchers were rubber-stamping their own work, circumventing peer review at five high-profile publishers.Scientific journals? Not exactly a badge of legitimacy, given that the International Journal of Advanced Computer Technology recently accepted for publication a paper titled “Get Me Off Your Fucking Mailing List,” whose text was nothing more than those seven words, repeated over and over for 10 pagesTwo other journals allowed an engineer posing as Maggie Simpson and Edna Krabappel to publish a paper, “Fuzzy, Homogeneous Configurations.”Revolutionary findings? Possibly fabricated. In May, a couple of University of California, Berkeley, grad students discovered irregularities in Michael LaCour’s influential paper suggesting that an in-person conversation with a gay person could change how people felt about same-sex marriage. The journal Science retracted the paper shortly after, when LaCour’s co-author could find no record of the data.

This article  was a really interesting read. For people interested studying the relation between empiricism, science, and Truth, it's worth checking out.

  • Upvote 3

Share this post


Link to post
Share on other sites

All the quote's links are flat-out fraud of one form or another.   They have nothing to do with failures of science.

 

Scientific fraud is nothing new (it goes back at least as far as piltdown man), but the technology involved in the first case certainly is.  (That particular kind of thing didn't happen even 10 years ago because there weren't these automated systems).  This is the only truly worrying one on there.

 

The next few links is more instances of fraud, but it isn't the submitters, it's the journal which is fraudulent.  These kinds of online pay-to-publish journals should not be considered part of the scientific record, and have no prestige.  They only appear to be science journals to people outside the field, and so aren't really an indict of any science.  (Indeed, no scientific fraud has been conducted because the frauds aren't scientists or part of the scientific community.  It's non-scientific fraud - and you'll note it is scientists exposing them).  So trying to push those off as instances of 'scientific journals' accepting ridiculous papers fails, because they aren't really scientific journals.  I could set up a website tomorrow, call it 'Journal of Phlobotinum Research', accept 'papers' for it, and publish them on the website - that's basically what those 'journals' are, and that does not make them part of the scientific record.

 

The last one isn't even science.  "Political Science" is not a science.  But the system worked as it should - people noticed irregularities in a study, tried to replicate the methods, and discovered the fraud. (And given the rest of the source blog's examples, it's pretty clear the writers aren't actually scientists, especially since their example doesn't even once mention a null hypothesis).

 

There will always be fraud, because there will always be people who feel the need to discover something groundbreaking when they didn't, or otherwise achieve recognition they haven't earned.  But because science does have to measure up to reality, fraud generally gets found out eventually.  (That needing to be falsifiable thing really comes in handy in the long term, even if we can get distracted by shucksters in the short term).

 

Now, the quoted article has a point - we need to be forthcoming about fraud and the reasons for paper retractions.  And retraction watch has had a hugely positive effect.  We also need to reward reviewers for the effort they put in to encourage them to be responsible reviewers.

 

Edit: Also, scientists getting to recommend reviewers is terrible editorial practice.  That sort of thing was really taking off about 10 years ago, and it was a terrible idea then, even before noticing the systems were gameable.  It's the editor's job to find qualified reviewers - passing that off to the submitting scientist is ethically dubious and terrible practice.

Edited by Squirrelloid
  • Upvote 2

Share this post


Link to post
Share on other sites

The last one isn't even science.  "Political Science" is not a science.  But the system worked as it should - people noticed irregularities in a study, tried to replicate the methods, and discovered the fraud. (And given the rest of the source blog's examples, it's pretty clear the writers aren't actually scientists, especially since their example doesn't even once mention a null hypothesis).

 

If you investigate that story, one thing you'll find is that the person who discovered the discrepancy was advised by all his fellow students and teachers and mentors to be quiet, to not pick a fight with someone more entrenched in the field than him, to sit tight and not speak up about potential fraud because it could ruin his career. He inexplicably came forward anyway, and has since said that he feels his story should be reassuring to people who worry about science being biased, since he met with no repercussions, but I think he might just be the exception that proves the rule. He is new to research, while the people advising him were old. Where did this widespread perception that honesty can get you into trouble come from, if not common experience? His story is far from the only place that such a perception can be found. It could just be that these tales are simply social science urban legends, and that definitely has something to do with it, but I don't think it can explain everything.

Edited by Chaos
  • Upvote 1

Share this post


Link to post
Share on other sites

If you investigate that story, one thing you'll find is that the person who discovered the discrepancy was advised by all his fellow students and teachers and mentors to be quiet, to not pick a fight with someone more entrenched in the field than him, to sit tight and not speak up about potential fraud because it could ruin his career. He inexplicably came forward anyway, and has since said that he feels his story should be reassuring to people who worry about science being biased, since he met with no repercussions, but I think he might just be the exception that proves the rule. He is new to research, while the people advising him were old. Where did this widespread perception that honesty can get you into trouble come from, if not common experience? His story is far from the only place that such a perception can be found. It could just be that these tales are simply social science urban legends, and that definitely has something to do with it, but I don't think it can explain everything.

 

Yeah, I'd actually read a detailed piece about him and the fraud months ago, iirc.

 

Accusations of fraud have always been treated with trepidation by academics because fraud cuts to the very heart of academic integrity.  To falsely accuse someone could do incalculable damage to their reputation and create a whiff of impropriety even after the accusation was dismissed.  I'm not sure 'career ruination' is the real reasoning, because i can't think of anyone's career who was ruined by accusing someone of fraud (although i haven't done exhaustive research), but those stories with admonishments of 'it could ruin your career' never come with *actual examples of careers ruined*.  I think the real motivation for caution is the worry that it isn't actually fraud, and basic empathy of an honest academic who understands how badly even an accusation of fraud could ruin him, so they caution others to be careful in accusing other academics of fraud.  

 

To some degree there's a point to that - you should have reasonable certainty that fraud was committed and go about acquiring actual evidence of fraud (as happened in this case).  And you should approach appropriate members of the academic community with that evidence before leveling a public accusation, to make sure other, preferably 'unbiased', observers agree with your conclusions from that data.  Because everyone would wish to have their own reputation given appropriate protections if they were the target of the accusation.  

Edited by Squirrelloid
  • Upvote 2

Share this post


Link to post
Share on other sites

Didn't see this until now.

 

There has been an explosion in the number of journals as the cost of publishing is low and the subscription fees are high which provides great financial incentives to the publisher. They have also recognized that there are more graduate school and more researchers than ever before - all of whom need space for publication.

 

It's currently a perfect storm of bad incentives. If people stop subscribing to hokey journals, the whole problem disappears.

Share this post


Link to post
Share on other sites

Didn't see this until now.

 

There has been an explosion in the number of journals as the cost of publishing is low and the subscription fees are high which provides great financial incentives to the publisher. They have also recognized that there are more graduate school and more researchers than ever before - all of whom need space for publication.

 

It's currently a perfect storm of bad incentives. If people stop subscribing to hokey journals, the whole problem disappears.

I don't think so - a lot of these bullshit journals make their money by making you pay to publish, not by subscription.

Share this post


Link to post
Share on other sites

I think this is a helpful reminder:

 

Human fallibilities send the scientific process hurtling in fits, starts and misdirections instead of in a straight line from question to truth.

 

Christie Aschwanden

 

The science we get is only as good as the character of the scientists.  Although it would be nice to have something that was more substantive.  A quote from the article would be sufficient to prove that warrant.

Share this post


Link to post
Share on other sites

A lot of the studies talking about failure to replicate look only at popular studies within supposedly reputable journals. I don't think hokey journals are the problem. Incentives are bad everywhere.

Whenever I think about or discuss this issue I usually just get really mad and want to execute all practitioners of poor research design, those who publish them, and all university PR teams everywhere. I know that's not a useful reaction. However, I've also never seen an argument for a solution that I think will actually work, so it's hard to be reasonable.

Edited by Chaos

Share this post


Link to post
Share on other sites

To be fair, I tend to think university PR departments may have it rough.  

 

Getting stuff covered may not be all that easy, especially for Tier 2 and Tier 3 universities competing against big universities in their market.

 

Also, academic research, except save the most applicable is probably not going to get covered.

 

And honestly some of this inability to replicate is about the beast that is research, not any ill will on the part of the researchers.  And the only way you can get peer review thats deep and wide is this marketing machine.

 

Not withstanding the above: the headlines in popular journal articles, which then get picked up by authors (popularizers) to sell their books is really, really annoying.

 

Sam Harris is guilty of this with respect to the Libet experiments.  The Libet experiments don't even get close to proving that free will isn't a think.  Not to mention science is meant to find causality--the opposite of free will.  Science looks for dominoes, not agency and intention.  Even though it sometimes find those popping up like fireworks amid all the rest of life.

Edited by nathan_debate
  • Upvote 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...