In order to ensure that this thread continues to meet TL standards and follows the proper guidelines, we will be enforcing the rules in the OP more strictly. Be sure to give them a re-read to refresh your memory! The vast majority of you are contributing in a healthy way, keep it up!
NOTE: When providing a source, explain why you feel it is relevant and what purpose it adds to the discussion if it's not obvious. Also take note that unsubstantiated tweets/posts meant only to rekindle old arguments can result in a mod action.
I think the panic about the crisis may be a bit premature. It should be looked at critically. Not used as some sort of final proof that mordern science is all bullshit.
Premature? There's been a steady (and huge) stream of increasing evidence for the past ~6 years about it. This isn't something that came up yesterday.
Even more damning is the theoretical evidence, tbh. The p < .05 publication bar (though there's recently a movement to change it) is comically low for the power of most experiments involving humans, and you can demonstrate that without even doing an experiment (using simulations on randomly generated assumptions). And that doesn't even take into account the all of the incentives (and options!) to nudge your results past that bar.
That article is mostly junk btw imo. I have respect for Nate Silver's work (as I've said before), but some of the other writers who publish on his site are pretty subpar imo.
If I had to make a comparison, I'd say a lot of the people claiming there isn't a replication crisis are doing some Donald Trump-type alternate reality-ism. And honestly, it makes a lot of sense to do so from their perspective. The replication crisis has the potential to destroy a lot of careers of influential academics, and they're acting like anyone else does when faced with that threat.
Hell, I worked with people from other departments when I was in graduate school who did a bunch of this stuff (multiple hypothesis testing, garden of forking paths, etc.) and didn't even know it was wrong. This stuff was ubiquitous until pretty recently.
Look up Brian Wansink. This guy was a "field leading" scientist out of Cornell (Ivy League!) academic who made a career out of this stuff, and only got caught because he was so ignorant of his pervasive methodological errors that he was exposed because he fucking blogged about it.
Obviously science is useful and we shouldn't stop doing it, but this is a real mess and it needs to be thoroughly cleaned up.
Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol.
I am not a statistician like our esteemed friend, mozoku, but having spent a little time in the lab, working with proteins and DNA, and trying to get reliable data, it was obvious to me a decade ago that reproducibility is a serious problem. I have spoken before about the some of the worst culprits (including diet science), and I agree with mozoku that that widely read 538 article is, in the words of slefin, not sound.
Now this is a convincing argument that doesn't revert back to a left vs right dynamic.
Is it a drop in quality due to increased volume? That there are more studies, causing more shit studies and less critical review? That by the time the previous study is proven to be poor, the group has secured funding for a new study?
I've seen this problem as well, particularly in psychology but even in basic pharmacological studies to an extent. A large contributor to the problem is the incentive to publish above all else and the corresponding low standards for doing research.
That said, you hit the nail on the head; Mozoku brings up a great point and is obviously knowledgeable on the the topic, but he brutally politicizes it to the point where it's hard to take him seriously. Bringing it up as a cudgel to try to de legitimize the use of science in political discussions (which he often does) is the dangerous type of false equivicating that hurts the integrity of political discourse in this country. It gives rise to dangerous strains of anti intellectualism, commonly from the right against the left.
The implication when this topic is brought up is a nihilistic one; that "the science is questionable, so all discourse is equal". This is intellectually lazy and completely ignores the epistemic value that our scientific process has to society, even if it is flawed.
I think the panic about the crisis may be a bit premature. It should be looked at critically. Not used as some sort of final proof that mordern science is all bullshit.
Premature? There's been a steady (and huge) stream of increasing evidence for the past ~6 years about it. This isn't something that came up yesterday.
Even more damning is the theoretical evidence, tbh. The p < .05 publication bar (though there's recently a movement to change it) is comically low for the power of most experiments involving humans, and you can demonstrate that without even doing an experiment (using simulations on randomly generated assumptions). And that doesn't even take into account the all of the incentives (and options!) to nudge your results past that bar.
That article is mostly junk btw imo. I have respect for Nate Silver's work (as I've said before), but some of the other writers who publish on his site are pretty subpar imo.
If I had to make a comparison, I'd say a lot of the people claiming there isn't a replication crisis are doing some Donald Trump-type alternate reality-ism. And honestly, it makes a lot of sense to do so from their perspective. The replication crisis has the potential to destroy a lot of careers of influential academics, and they're acting like anyone else does when faced with that threat.
Hell, I worked with people from other departments when I was in graduate school who did a bunch of this stuff (multiple hypothesis testing, garden of forking paths, etc.) and didn't even know it was wrong. This stuff was ubiquitous until pretty recently.
Look up Brian Wansink. This guy was a "field leading" scientist out of Cornell (Ivy League!) academic who made a career out of this stuff, and only got caught because he was so ignorant of his pervasive methodological errors that he was exposed because he fucking blogged about it.
Obviously science is useful and we shouldn't stop doing it, but this is a real mess and it needs to be thoroughly cleaned up.
Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol.
I am not a statistician like our esteemed friend, mozoku, but having spent a little time in the lab, working with proteins and DNA, and trying to get reliable data, it was obvious to me a decade ago that reproducibility is a serious problem. I have spoken before about the some of the worst culprits (including diet science), and I agree with mozoku that that widely read 538 article is, in the words of slefin, not sound.
Now this is a convincing argument that doesn't revert back to a left vs right dynamic.
Is it a drop in quality due to increased volume? That there are more studies, causing more shit studies and less critical review? That by the time the previous study is proven to be poor, the group has secured funding for a new study?
What Stratos_Spear said above about some of the causes is accurate.
I'm not sure the quality is actually lower, or if it was just always low. Most published studies probably don't yield replication attempts, and failed replications usually aren't published or even communicated to the original author in many (most?) cases. There's hasn't been (and still isn't really) a real system in place for cleaning up much of the literature once it's out there, so I'm not sure if we know if today's publications are worse than the past.
A lot of scientists are pretty skeptical about the quality of the peer review process. There's really not any incentive as a peer reviewer to do a thorough and meaningful job reviewing, as replication attempts are relatively rare and nobody really blames the peer reviewer for the failed replication of reasonable-ish publication. There's not really any benefit to even doing peer reviews. It's generally more productive for one's career to spend his time on his own research than spending huge amounts of time on peer review. And in fairness, a peer reviewer is a single human being--it's a hard job and they're going to make mistakes.
Funding is usually given to those with publications in journals (the more well-known, the better), and isn't really correlated with replication rates (as published replication attempts are rare anyway). Well-known journals have a definite skew towards certain types of papers: lots of authors, grand-sounding claims, and novelty (i.e. not replications). As far as I'm aware, the replication rates of well-known journals aren't significantly better than less well-known journals.
There's a statistical element as well. At this point, pretty much the whole community agrees that p < 0.05 was a poor bar for publication. Beyond that, there's a philosophical divide on whether the Null Hypothesis Significance Testing (NHST) was ever a good idea in the first place, and, in each group, there's about as many opinions on the best way to go forward as there are statisticians.
In general, statisticians weren't aware of how big the problem with NHST p < 0.05 paradigm was until fairly recently. As recently as 2007-2010, the leading statisticians in confronting the replication crisis were routinely missing errors that are obvious to anyone who's paid any attention to the replication crisis these days. I'm not entirely sure why this is, but my guess is that the lack of published replications meant there was no real feedback mechanism to force them to think about the issues. One possible reason why they started thinking about this all of a sudden is the widespread adoption of computers and simulations in statistics, which makes it relatively easy for anyone to test these sorts of things in an less than hour's time--but this is admittedly entirely my speculation.
I think the panic about the crisis may be a bit premature. It should be looked at critically. Not used as some sort of final proof that mordern science is all bullshit.
Premature? There's been a steady (and huge) stream of increasing evidence for the past ~6 years about it. This isn't something that came up yesterday.
Even more damning is the theoretical evidence, tbh. The p < .05 publication bar (though there's recently a movement to change it) is comically low for the power of most experiments involving humans, and you can demonstrate that without even doing an experiment (using simulations on randomly generated assumptions). And that doesn't even take into account the all of the incentives (and options!) to nudge your results past that bar.
That article is mostly junk btw imo. I have respect for Nate Silver's work (as I've said before), but some of the other writers who publish on his site are pretty subpar imo.
If I had to make a comparison, I'd say a lot of the people claiming there isn't a replication crisis are doing some Donald Trump-type alternate reality-ism. And honestly, it makes a lot of sense to do so from their perspective. The replication crisis has the potential to destroy a lot of careers of influential academics, and they're acting like anyone else does when faced with that threat.
Hell, I worked with people from other departments when I was in graduate school who did a bunch of this stuff (multiple hypothesis testing, garden of forking paths, etc.) and didn't even know it was wrong. This stuff was ubiquitous until pretty recently.
Look up Brian Wansink. This guy was a "field leading" scientist out of Cornell (Ivy League!) academic who made a career out of this stuff, and only got caught because he was so ignorant of his pervasive methodological errors that he was exposed because he fucking blogged about it.
Obviously science is useful and we shouldn't stop doing it, but this is a real mess and it needs to be thoroughly cleaned up.
Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol.
I am not a statistician like our esteemed friend, mozoku, but having spent a little time in the lab, working with proteins and DNA, and trying to get reliable data, it was obvious to me a decade ago that reproducibility is a serious problem. I have spoken before about the some of the worst culprits (including diet science), and I agree with mozoku that that widely read 538 article is, in the words of slefin, not sound.
Now this is a convincing argument that doesn't revert back to a left vs right dynamic.
Is it a drop in quality due to increased volume? That there are more studies, causing more shit studies and less critical review? That by the time the previous study is proven to be poor, the group has secured funding for a new study?
the short answer is that, in the example of testing protein activity, somewhere between growing the protein, lysing the cells, purifying the protein, storing the protein, thawing the protein, and testing the protein under different conditions, it became nearly impossible to get really consistent data points. every single step of the process can introduce variance (or error). when you consider that a lot of these studies are done on a budget, in a small lab, with only a handful of people (if that) doing all these things, you can only really talk about trends and broad relative differences across labs for a lot of these mutants. sure you can get an accurate DNA sequence and you have a pretty accurate understanding about a lot of these things, but when it comes to actually predicting how any particular sample will do, its very very difficult to get consistent results. that's why grad students are running hundreds of experiments and throwing out dozens of the ones they do. the error bars are huge. and then people write papers for publication and in the methods section only describe half a dozen variables, when the number of things impacting the outcome is many multiples of that. i can't tell you how many times i've read a chemistry paper where the methods section barely tells you anything about the materials compositions, where they got the materials, what they did in between experiments, what scale they were working at, what the particle size was, etc. and that's chemistry/materials science, where the reproducibility is fairly high compared to other sciences.
The reproducibility issue is also influenced by how long the experiments take. Whenever you're dealing with living things, the studies can run for months (even years!). As a PhD student, if you get "unlucky" and your experiments do not yield very conclusive results, then there is a lot of pressure to get at least "something" publishable or the PhD student might not even graduate. It doesn't help that faculty are absolutely pressured themselves to publish or forget about getting any funding.
I should say that in my field (materials science), most articles in high-quality journals are reproducible. Experiments in this field generally don't take more than a week or two, so reproducing the newest experiment/recipe from a science/nature paper is "easy" and helps yourself progress.
On December 30 2017 11:32 mozoku wrote: Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol. No wonder, considering they don't foot the bill for their own product.
I'm in the arts so I can't usefully comment on the science stuff, but our publishing is similar enough for me to comment on this, and it absolutely baffles me. The estimates I've heard for commercial outfits gives them profit at around 35% (though there are claims of up to 50%). How have we been hoodwinked into allowing a massive industry to parasitically leech off research they don't pay for, reviewed by people they don't pay for, and sold to the very people who have just done all the work without pay so that they can use it to educate and produce more work that the publishing industry won't pay for but will sell back to them?
In short: why do commercial publishers exist?
Possibly because nobody ways to pay a couple of hundred bucks to get published. (Full disclosure: it's been a while since I checked the actual numbers.) There are a bunch of open source journals. Just with little exposure and - what's the index called again... Impact rating? - so publishing in nature will gain you more exposure and "impact" than publishing in sustainability. Or possibly there aren't any in your field. Or the American universities not giving a damn because they have access to most of em and it's always been that way.
It's in part a decision by those who publish to fuel the fire by publishing in those fucking expensive journals instead of the open ones. But departments being at risk of getting funding cut for insufficient high profile publications is not helping at al. I find this especially ludicrous because a lot of research here in Germany is state funded (but as the US's education system appears to be for profit as well, idk how that fares on comparison) so you basically have to pay twice. First for finding research and afterwards for reading the published findings.
On December 30 2017 11:32 mozoku wrote: Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol. No wonder, considering they don't foot the bill for their own product.
I'm in the arts so I can't usefully comment on the science stuff, but our publishing is similar enough for me to comment on this, and it absolutely baffles me. The estimates I've heard for commercial outfits gives them profit at around 35% (though there are claims of up to 50%). How have we been hoodwinked into allowing a massive industry to parasitically leech off research they don't pay for, reviewed by people they don't pay for, and sold to the very people who have just done all the work without pay so that they can use it to educate and produce more work that the publishing industry won't pay for but will sell back to them?
In short: why do commercial publishers exist?
My understanding is that there lacks a better alternative. Some promise with alternatives like online / open access journals, but many are just pay to play dumpster fires. Until that changes there's nothing to push down their margins.
Potholer had a good vid on crap vs legitimate journals a few months back:
I know it's buzzfeed, but the material itself is just pure candy. That being said, the editor's cynicism in working with someone like that, being perfectly aware of how full of it he is is painful to witness.
Trump and his goons were the perfect targets for foreign intelligence, because of their vices. This also speaks to the fact that Mueller’s investigation is not just criminal, it’s also counterintelligence. Trump is now saying collusion is not illegal, but that’s not the whole story.
During a night of heavy drinking at an upscale London bar in May 2016, George Papadopoulos, a young foreign policy adviser to the Trump campaign, made a startling revelation to Australia’s top diplomat in Britain: Russia had political dirt on Hillary Clinton.
About three weeks earlier, Mr. Papadopoulos had been told that Moscow had thousands of emails that would embarrass Mrs. Clinton, apparently stolen in an effort to try to damage her campaign.
Exactly how much Mr. Papadopoulos said that night at the Kensington Wine Rooms with the Australian, Alexander Downer, is unclear. But two months later, when leaked Democratic emails began appearing online, Australian officials passed the information about Mr. Papadopoulos to their American counterparts, according to four current and former American and foreign officials with direct knowledge of the Australians’ role.
The hacking and the revelation that a member of the Trump campaign may have had inside information about it were driving factors that led the F.B.I. to open an investigation in July 2016 into Russia’s attempts to disrupt the election and whether any of President Trump’s associates conspired.
If Mr. Papadopoulos, who pleaded guilty to lying to the F.B.I. and is now a cooperating witness, was the improbable match that set off a blaze that has consumed the first year of the Trump administration, his saga is also a tale of the Trump campaign in miniature. He was brash, boastful and underqualified, yet he exceeded expectations. And, like the campaign itself, he proved to be a tantalizing target for a Russian influence operation.
While some of Mr. Trump’s advisers have derided him an insignificant campaign volunteer or a “coffee boy,” interviews and new documents show that he stayed influential throughout the campaign. Two months before the election, for instance, he helped arrange a New York meeting between Mr. Trump and President Abdel Fattah el-Sisi of Egypt.
The information that Mr. Papadopoulos gave to the Australians answers one of the lingering mysteries of the past year: What so alarmed American officials to provoke the F.B.I. to open a counterintelligence investigation into the Trump campaign months before the presidential election?
It was not, as Mr. Trump and other politicians have alleged, a dossier compiled by a former British spy hired by a rival campaign. Instead, it was firsthand information from one of America’s closest intelligence allies.
On December 30 2017 19:03 EnDeR_ wrote: The reproducibility issue is also influenced by how long the experiments take. Whenever you're dealing with living things, the studies can run for months (even years!). As a PhD student, if you get "unlucky" and your experiments do not yield very conclusive results, then there is a lot of pressure to get at least "something" publishable or the PhD student might not even graduate. It doesn't help that faculty are absolutely pressured themselves to publish or forget about getting any funding.
I should say that in my field (materials science), most articles in high-quality journals are reproducible. Experiments in this field generally don't take more than a week or two, so reproducing the newest experiment/recipe from a science/nature paper is "easy" and helps yourself progress.
Huh. I never realized some places/fields required actual publication as a graduation requirement rather than just journal submission. Glad I am where I am!
On December 30 2017 11:32 mozoku wrote: Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol. No wonder, considering they don't foot the bill for their own product.
I'm in the arts so I can't usefully comment on the science stuff, but our publishing is similar enough for me to comment on this, and it absolutely baffles me. The estimates I've heard for commercial outfits gives them profit at around 35% (though there are claims of up to 50%). How have we been hoodwinked into allowing a massive industry to parasitically leech off research they don't pay for, reviewed by people they don't pay for, and sold to the very people who have just done all the work without pay so that they can use it to educate and produce more work that the publishing industry won't pay for but will sell back to them?
In short: why do commercial publishers exist?
What mozoku conveniently forgets to mention is that much like shitty for profit colleges no one takes shitty for profit journals or research mills seriously, except amateurs and laymen.
On December 30 2017 19:03 EnDeR_ wrote: The reproducibility issue is also influenced by how long the experiments take. Whenever you're dealing with living things, the studies can run for months (even years!). As a PhD student, if you get "unlucky" and your experiments do not yield very conclusive results, then there is a lot of pressure to get at least "something" publishable or the PhD student might not even graduate. It doesn't help that faculty are absolutely pressured themselves to publish or forget about getting any funding.
I should say that in my field (materials science), most articles in high-quality journals are reproducible. Experiments in this field generally don't take more than a week or two, so reproducing the newest experiment/recipe from a science/nature paper is "easy" and helps yourself progress.
haha, in undergrad i spent about a year trying to produce a certain protein. problem was that the protein was apparently toxic in some way, and it kept killing the bacterial expression vectors so i could never get usable amounts for analysis. good times.
On December 30 2017 11:32 mozoku wrote: Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol. No wonder, considering they don't foot the bill for their own product.
I'm in the arts so I can't usefully comment on the science stuff, but our publishing is similar enough for me to comment on this, and it absolutely baffles me. The estimates I've heard for commercial outfits gives them profit at around 35% (though there are claims of up to 50%). How have we been hoodwinked into allowing a massive industry to parasitically leech off research they don't pay for, reviewed by people they don't pay for, and sold to the very people who have just done all the work without pay so that they can use it to educate and produce more work that the publishing industry won't pay for but will sell back to them?
In short: why do commercial publishers exist?
What mozoku conveniently forgets to mention is that much like shitty for profit colleges no one takes shitty for profit journals or research mills seriously, except amateurs and laymen.
I think you're confusing for-profit journals with pay-to-publish journals.
Even if we we're to assume you were right, for the sake of argument:
How exactly are you, as a supposed pro-science and pro-education liberal, trying to argue that requiring the public to pay large sums to private companies to view research completely funded by public money is good for either the advancement of either science or education?
On December 30 2017 11:32 mozoku wrote: Another hoot is the $25B annual revenue for-profit journal industry that charges people to read papers written by academics (paid by universities) that do experiments funded by tax-payers--and their margins are better then Apple lol. No wonder, considering they don't foot the bill for their own product.
I'm in the arts so I can't usefully comment on the science stuff, but our publishing is similar enough for me to comment on this, and it absolutely baffles me. The estimates I've heard for commercial outfits gives them profit at around 35% (though there are claims of up to 50%). How have we been hoodwinked into allowing a massive industry to parasitically leech off research they don't pay for, reviewed by people they don't pay for, and sold to the very people who have just done all the work without pay so that they can use it to educate and produce more work that the publishing industry won't pay for but will sell back to them?
In short: why do commercial publishers exist?
What mozoku conveniently forgets to mention is that much like shitty for profit colleges no one takes shitty for profit journals or research mills seriously, except amateurs and laymen.
I think you're confusing for-profit journals with pay-to-publish journals.
Even if we we're to assume you were right, for the sake of argument:
How exactly are you, as a supposed pro-science and pro-education liberal, trying to argue that requiring the public to pay large sums to private companies to view research completely funded by public money is good for either the advancement of either science or education?
I don't think he is arguing that or ever claimed that. at least I don't recall him saying that, or anything that could be reasonably construed to be that. can you point to where you think he argued that?
EPT
