2023-08-01 Tue 12:36 [[Gilbert 2016 - Comment on “Estimating the reproducibility of psychological science”.pdf]] >%% >```annotation-json >{"created":"2023-08-01T19:40:28.412Z","updated":"2023-08-01T19:40:28.412Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":1287,"end":1544},{"type":"TextQuoteSelector","exact":"Here, we showthat when these results are corrected for error,power, and bias, they provide no support for thisconclusion. In fact, the data are consistent withthe opposite conclusion, namely, that the repro-ducibility of psychological science is quite high.","prefix":"s”in psychological science (2). ","suffix":"First, we will discuss the issue"}]}]} >``` >%% >*%%PREFIX%%s”in psychological science (2).%%HIGHLIGHT%% ==Here, we showthat when these results are corrected for error,power, and bias, they provide no support for thisconclusion. In fact, the data are consistent withthe opposite conclusion, namely, that the repro-ducibility of psychological science is quite high.== %%POSTFIX%%First, we will discuss the issue* >%%LINK%%[[#^jqnfsr11wnk|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^jqnfsr11wnk >%% >```annotation-json >{"created":"2023-08-01T19:43:15.756Z","updated":"2023-08-01T19:43:15.756Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":2782,"end":3390},{"type":"TextQuoteSelector","exact":"many of OSC’s replication studiesdrew their samples from different populationsthan the original studies did. An original study thatmeasured American’s attitudes toward African-Americans (3) was replicated with Italians, whodo not share the same stereotypes; an originalstudy that asked college students to imagine beingcalled on by a professor (4) was replicated withparticipants who had never been to college; andan original study that asked students who com-mute to school to choose between apartments thatwere short and long drives from campus (5) was replicated with students who do not commute toschool.","prefix":"ther ways as well. For example, ","suffix":" What’s more, many of OSC’s repl"}]}]} >``` >%% >*%%PREFIX%%ther ways as well. For example,%%HIGHLIGHT%% ==many of OSC’s replication studiesdrew their samples from different populationsthan the original studies did. An original study thatmeasured American’s attitudes toward African-Americans (3) was replicated with Italians, whodo not share the same stereotypes; an originalstudy that asked college students to imagine beingcalled on by a professor (4) was replicated withparticipants who had never been to college; andan original study that asked students who com-mute to school to choose between apartments thatwere short and long drives from campus (5) was replicated with students who do not commute toschool.== %%POSTFIX%%What’s more, many of OSC’s repl* >%%LINK%%[[#^ym76cmzemj|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^ym76cmzemj >%% >```annotation-json >{"created":"2023-08-01T19:50:39.447Z","updated":"2023-08-01T19:50:39.447Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":4177,"end":4291},{"type":"TextQuoteSelector","exact":"All of these infidelities are potential sources ofrandom error that the OSC’s benchmark did not take into account.","prefix":"-tially zero in all conditions).","suffix":" So how many of their replica-ti"}]}]} >``` >%% >*%%PREFIX%%-tially zero in all conditions).%%HIGHLIGHT%% ==All of these infidelities are potential sources ofrandom error that the OSC’s benchmark did not take into account.== %%POSTFIX%%So how many of their replica-ti* >%%LINK%%[[#^02h73kl8ir5u|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^02h73kl8ir5u >%% >```annotation-json >{"created":"2023-08-01T19:52:13.314Z","updated":"2023-08-01T19:52:13.314Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":5371,"end":5625},{"type":"TextQuoteSelector","exact":"Applying this esti-mate to OSC’s data produces a sobering conclusion: If every one of the 100 original studies that OSCattempted to replicate had described a true ef-fect, then more than 34 of their replication studiesshould have failed by chance alone. ","prefix":"rvals of the “original effects.”","suffix":"[All informa-tion and code neces"}]}]} >``` >%% >*%%PREFIX%%rvals of the “original effects.”%%HIGHLIGHT%% ==Applying this esti-mate to OSC’s data produces a sobering conclusion: If every one of the 100 original studies that OSCattempted to replicate had described a true ef-fect, then more than 34 of their replication studiesshould have failed by chance alone.== %%POSTFIX%%[All informa-tion and code neces* >%%LINK%%[[#^jha78d0vao7|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^jha78d0vao7 >%% >```annotation-json >{"created":"2023-08-01T19:52:31.619Z","updated":"2023-08-01T19:52:31.619Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":5720,"end":5932},{"type":"TextQuoteSelector","exact":"The bottom lineis that OSC allowed considerable infidelities thatintroduced random error and decreased the rep-lication rate but then compared their results toa benchmark that did not take this error intoaccount.","prefix":"e archived in Dataverse ( 10).] ","suffix":"Second, we will discuss the issu"}]}]} >``` >%% >*%%PREFIX%%e archived in Dataverse ( 10).]%%HIGHLIGHT%% ==The bottom lineis that OSC allowed considerable infidelities thatintroduced random error and decreased the rep-lication rate but then compared their results toa benchmark that did not take this error intoaccount.== %%POSTFIX%%Second, we will discuss the issu* >%%LINK%%[[#^6vkslmaurre|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^6vkslmaurre >%% >```annotation-json >{"created":"2023-08-01T19:53:23.158Z","updated":"2023-08-01T19:53:23.158Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":6229,"end":6843},{"type":"TextQuoteSelector","exact":"In contrast, MLP attempted toreplicate each of its studies 35 or 36 times andthen pooled the data. MLP’s much more pow-erful method produced a much more hearten-ing result: A full 85% of the original studies weresuccessfully replicated. What would have hap-pened to MLP’s heartening result if they hadused OSC’s method? Of MLP’s 574 replicationstudies, only 195 produced effects that fell withinthe confidence interval of the original, publishedstudy. In other words, if MLP had used OSC’smethod, they would have reported an unsettlingreplication rate of 34% rather than the hearten-ing 85% they actually reported.","prefix":"interval of theoriginal study). ","suffix":" (A similar resultoccurs when we"}]}]} >``` >%% >*%%PREFIX%%interval of theoriginal study).%%HIGHLIGHT%% ==In contrast, MLP attempted toreplicate each of its studies 35 or 36 times andthen pooled the data. MLP’s much more pow-erful method produced a much more hearten-ing result: A full 85% of the original studies weresuccessfully replicated. What would have hap-pened to MLP’s heartening result if they hadused OSC’s method? Of MLP’s 574 replicationstudies, only 195 produced effects that fell withinthe confidence interval of the original, publishedstudy. In other words, if MLP had used OSC’smethod, they would have reported an unsettlingreplication rate of 34% rather than the hearten-ing 85% they actually reported.== %%POSTFIX%%(A similar resultoccurs when we* >%%LINK%%[[#^bhnb0vlo3yg|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^bhnb0vlo3yg >%% >```annotation-json >{"created":"2023-08-01T19:54:06.286Z","updated":"2023-08-01T19:54:06.286Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":7077,"end":7457},{"type":"TextQuoteSelector","exact":"Third, we will discuss the issue of bias. Theforegoing analyses generously assume that infi-delities are a source of random error that areequally likely to increase or decrease the likeli-hood of successful replication. Is this assumptiontrue, or were the infidelities in OSC’s replication studies more likely to decrease than to increase the likelihood of successful replication?","prefix":"s the actualrate of replication.","suffix":" Answering thisquestion requires"}]}]} >``` >%% >*%%PREFIX%%s the actualrate of replication.%%HIGHLIGHT%% ==Third, we will discuss the issue of bias. Theforegoing analyses generously assume that infi-delities are a source of random error that areequally likely to increase or decrease the likeli-hood of successful replication. Is this assumptiontrue, or were the infidelities in OSC’s replication studies more likely to decrease than to increase the likelihood of successful replication?== %%POSTFIX%%Answering thisquestion requires* >%%LINK%%[[#^ignuxl8yt6|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^ignuxl8yt6 >%% >```annotation-json >{"created":"2023-08-01T19:55:11.415Z","updated":"2023-08-01T19:55:11.415Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":7617,"end":7700},{"type":"TextQuoteSelector","exact":"OSC asked the authors of the originalstudy whether they endorsed the methodological","prefix":"nducting each replicationstudy, ","suffix":"RESEARCH1037-a1Harvard Universit"}]}]} >``` >%% >*%%PREFIX%%nducting each replicationstudy,%%HIGHLIGHT%% ==OSC asked the authors of the originalstudy whether they endorsed the methodological== %%POSTFIX%%RESEARCH1037-a1Harvard Universit* >%%LINK%%[[#^vzcnlo5qw6|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^vzcnlo5qw6 >%% >```annotation-json >{"created":"2023-08-01T19:55:19.830Z","updated":"2023-08-01T19:55:19.830Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":7950,"end":7995},{"type":"TextQuoteSelector","exact":"protocol for the to-be-attempted replication.","prefix":" MARCH 2016 • VOL 351 ISSUE 6277","suffix":" Only69% of the original authors"}]}]} >``` >%% >*%%PREFIX%%MARCH 2016 • VOL 351 ISSUE 6277%%HIGHLIGHT%% ==protocol for the to-be-attempted replication.== %%POSTFIX%%Only69% of the original authors* >%%LINK%%[[#^7bc9ehm9bin|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^7bc9ehm9bin >%% >```annotation-json >{"created":"2023-08-01T19:55:30.179Z","updated":"2023-08-01T19:55:30.179Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":8658,"end":8894},{"type":"TextQuoteSelector","exact":"When we compared the replication rates of theendorsed and unendorsed protocols, we discoveredthat the endorsed protocols were nearly four timesas likely to produce a successful replication (59.7%)as were the unendorsed protocols (15.4%)","prefix":"t does that indi-cator indicate?","suffix":". Thisstrongly suggests that the"}]}]} >``` >%% >*%%PREFIX%%t does that indi-cator indicate?%%HIGHLIGHT%% ==When we compared the replication rates of theendorsed and unendorsed protocols, we discoveredthat the endorsed protocols were nearly four timesas likely to produce a successful replication (59.7%)as were the unendorsed protocols (15.4%)== %%POSTFIX%%. Thisstrongly suggests that the* >%%LINK%%[[#^h9l4c9uqlnp|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^h9l4c9uqlnp >%% >```annotation-json >{"created":"2023-08-01T20:39:08.824Z","updated":"2023-08-01T20:39:08.824Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":9666,"end":9754},{"type":"TextQuoteSelector","exact":"SC’s data clearly provideno evidence for a “replication crisis”in psycholog-ical science","prefix":"ugh thatseems rather unlikely, O","suffix":".We applaud efforts to improve p"}]}]} >``` >%% >*%%PREFIX%%ugh thatseems rather unlikely, O%%HIGHLIGHT%% ==SC’s data clearly provideno evidence for a “replication crisis”in psycholog-ical science== %%POSTFIX%%.We applaud efforts to improve p* >%%LINK%%[[#^7b39o0xoz4r|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^7b39o0xoz4r >%% >```annotation-json >{"created":"2023-08-01T20:39:20.633Z","updated":"2023-08-01T20:39:20.633Z","document":{"title":"Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","link":[{"href":"urn:x-pdf:54a5c9e963816621341d32f9b80f48b0"},{"href":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf"}],"documentFingerprint":"54a5c9e963816621341d32f9b80f48b0"},"uri":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","target":[{"source":"vault:/z.Attachments/-Papers and books/Gilbert 2016 - Comment on %E2%80%9CEstimating the reproducibility of psychological science%E2%80%9D.pdf","selector":[{"type":"TextPositionSelector","start":9937,"end":9988},{"type":"TextQuoteSelector","exact":"metascienceis not exempt from the rules of science.","prefix":"at went into producing OSC. But ","suffix":" OSC useda benchmark that did no"}]}]} >``` >%% >*%%PREFIX%%at went into producing OSC. But%%HIGHLIGHT%% ==metascienceis not exempt from the rules of science.== %%POSTFIX%%OSC useda benchmark that did no* >%%LINK%%[[#^7e6qgsyolow|show annotation]] >%%COMMENT%% > >%%TAGS%% > ^7e6qgsyolow