Showing posts with label science. Show all posts
Showing posts with label science. Show all posts

Friday, August 16, 2019

16/8/19: Post-Millennials and the falling trust in institutions of coercion


A neat chart from Pew Research highlighting shifting demographics behind the changing trends in the U.S. public trust in core institutions:

Source: https://www.people-press.org/2019/07/22/how-americans-see-problems-of-trust/

Overall, the generational shift is in the direction of younger GenZ putting more trust in scientists and academics, as well as journalists, compared to previous generations; and less trust in military, police, religious leaders and business leaders. Notably, elected officials have pretty much low trust across all three key demographics.

Friday, June 16, 2017

16/6/17: Replicating Scientific Research: Ugly Truth


Continuing with the theme on 'What I've been reading lately?', here is a smashing paper on 'accuracy' of empirical economic studies.

The paper, authored by Hou, Kewei and Xue, Chen and Zhang, Lu, and titled "Replicating Anomalies" (most recent version is from June 12, 2017, but it is also available in an earlier version via NBER) effectively blows a whistle on what is going on in empirical research in economics and finance. Per authors, the vast literature that detects financial markets anomalies (or deviations away from the efficient markets hypothesis / economic rationality) "is infested with widespread p-hacking".

What's p-hacking? Well, it's a shady practice whereby researchers manipulate (by selective inclusion or exclusion) sample criteria (which data points to exclude from estimation) and test procedures (including model specifications and selective reporting of favourable test results), until insignificant results become significant. In other words, under p-hacking, researchers attempt to superficially maximise model and explanatory variables significance, or, put differently, they attempt to achieve results that confirm their intuition or biases.

What's anomalies? Anomalies are departures in the markets (e.g. in share prices) from the predictions generated by the models consistent with rational expectations and the efficient markets hypothesis. In other words, anomalies occur when markets efficiency fails.

There are scores of anomalies detected in the academic literature, prompting many researchers to advocate abandonment (in all its forms, weak and strong) of the idea that markets are efficient.

Hou, Xue and Zhang take these anomalies to the test. The compile "a large data library with 447 anomalies". The authors then control for a key problem with data across many studies: microcaps. Microcaps - or small capitalization firms - are numerous in the markets (accounting for roughly 60% of all stocks), but represent only 3% of total market capitalization. This is true for key markets, such as NYSE, Amex and NASDAQ. Yet, as authors note, evidence shows that microcaps "not only have the highest equal-weighted returns, but also the largest cross-sectional standard deviations in returns and anomaly variables among microcaps, small stocks, and big stocks." In other words, these are higher risk, higher return class of securities. Yet, despite this, "many studies overweight microcaps with equal-weighted returns, and often together with NYSE-Amex-NASDAQ breakpoints, in portfolio sorts." Worse, many (hundreds) of studies use 1970s regression technique that actually assigns more weight to microcaps. In simple terms, microcaps are the most common outlier and despite this they are given either same weight in analysis as non-outliers or their weight is actually elevated relative to normal assets, despite the fact that microcaps have little meaning in driving the actual markets (their weight in the total market is just about 3% in total).

So the study corrects for these problems and finds that, once microcaps are accounted for, the grand total of 286 anomalies (64% of all anomalies studied), and under more strict statistical signifcance test 380 (of 85% of all anomalies) "including 95 out of 102 liquidity variables (93%) are insignificant at the 5% level." In other words, the original studies claims that these anomalies were significant enough to warrant rejection of markets efficiency were not true when one recognizes one basic and simple problem with the data. Worse, per authors, "even for the 161 significant anomalies, their magnitudes are often much lower than originally reported. Among the 161, the q-factor model leaves 115 alphas insignificant (150 with t < 3)."

This is pretty damning for those of us who believe, based on empirical results published over the years, that markets are bounded-efficient, and it is outright savaging for those who claim that markets are perfectly inefficient. But, this tendency of researchers to silverplate statistics is hardly new.

Hou, Xue and Zhang provide a nice summary of research on p-hacking and non-replicability of statistical results across a range of fields. It is worth reading, because it dents significantly ones confidence in the quality of peer review and the quality of scientific research.

As the authors note, "in economics, Leamer (1983) exposes the fragility of empirical results to small specification changes, and proposes to “take the con out of econometrics” by reporting extensive sensitivity analysis to show how key results vary with perturbations in regression specification and in functional form." The latter call was never implemented in the research community.

"In an influential study, Dewald, Thursby, and Anderson (1986) attempt to replicate empirical results published at Journal of Money, Credit, and Banking [a top-tier journal], and find that inadvertent errors are so commonplace that the original results often cannot be reproduced."

"McCullough and Vinod (2003) report that nonlinear maximization routines from different software packages often produce very different estimates, and many articles published at American Economic Review [highest rated journal in economics] fail to test their solutions across different software packages."

"Chang and Li (2015) report a success rate of less than 50% from replicating 67 published papers from 13 economics journals, and Camerer et al. (2016) show a success rate of 61% from replicating 18 studies in experimental economics."

"Collecting more than 50,000 tests published in American Economic Review, Journal of Political Economy, and Quarterly Journal of Economics, [three top rated journals in economics] Brodeur, L´e, Sangnier, and Zylberberg (2016) document a troubling two-humped pattern of test statistics. The pattern features a first hump with high p-values, a sizeable under-representation of p-values just above 5%, and a second hump with p-values slightly below 5%. The evidence indicates p-hacking that authors search for specifications that deliver just-significant results and ignore those that give just-insignificant results to make their work more publishable."

If you think this phenomena is encountered only in economics and finance, think again. Here are some findings from other ' hard science' disciplines where, you know, lab coats do not lie.

"...replication failures have been widely documented across scientific disciplines in the past decade. Fanelli (2010) reports that “positive” results increase down the hierarchy of sciences, with hard sciences such as space science and physics at the top and soft sciences such as psychology, economics, and business at the bottom. In oncology, Prinz, Schlange, and Asadullah (2011) report that scientists at Bayer fail to reproduce two thirds of 67 published studies. Begley and Ellis (2012) report that scientists at Amgen attempt to replicate 53 landmark studies in cancer research, but reproduce the original results in only six. Freedman, Cockburn, and Simcoe (2015) estimate the economic costs of irreproducible preclinical studies amount to about 28 billion dollars in the U.S. alone. In psychology, Open Science Collaboration (2015), which consists of about 270 researchers, conducts replications of 100 studies published in top three academic journals, and reports a success rate of only 36%."

Let's get down to real farce: everyone in sciences knows the above: "Baker (2016) reports that 80% of the respondents in a survey of 1,576 scientists conducted by Nature believe that there exists a reproducibility crisis in the published scientific literature. The surveyed scientists cover diverse fields such as chemistry, biology, physics and engineering, medicine, earth sciences, and others. More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than 50% have failed to reproduce their own experiments. Selective reporting, pressure to publish, and poor use of statistics are three leading causes."

Yeah, you get the idea: you need years of research, testing, re-testing and, more often then not, you get the results are not significant or weakly significant. Which means that after years of research you end up with unpublishable paper (no journal would welcome a paper without significant results, even though absence of evidence is as important in science as evidence of presence), no tenure, no job, no pension, no prospect of a career. So what do you do then? Ah, well... p-hack the shit out of data until the editor is happy and the referees are satisfied.

Which, for you, the reader, should mean the following: when we say that 'scientific research established fact A' based on reputable journals publishing high quality peer reviewed papers on the subject, know that around half of the findings claimed in these papers, on average, most likely cannot be replicated or verified. And then remember, it takes one or two scientists to turn the world around from believing (based on scientific consensus at the time) that the Earth is flat and is the centre of the Universe, to believing in the world as we know it to be today.


Full link to the paper: Charles A. Dice Center Working Paper No. 2017-10; Fisher College of Business Working Paper No. 2017-03-010. Available at SSRN: https://ssrn.com/abstract=2961979.

Thursday, January 9, 2014

9/1/2014: Public Lecture in Biochemisty and Immunology


Fascinating stuff:

The School of Biochemistry & Immunology in TCD:  public lecture in Biomedical Frontiers series. 

Professor Ken Mok, "Who da mule? - Smuggling molecules across (biological) borders
" 

Key topics:
  • Why is basic research important? 
  • Can we really predict which 'horses' (= specific application areas) to bet on in the long run? 
  • Telling "a factual story in protein folding/misfolding research where a potential drug-transporting 'mule' - rather than 'horse' - was serendipitously found through basic studies. Widespread interest in this protein-fatty acid complex is growing due to its remarkable properties of selectively killing cancer cells while leaving healthy, differentiated cells intact."

Event details
:
Date: Wednesday 15th January  
       
Time: 6:30pm
       
Venue: Stanley Quek Theatre, Trinity Biomedical Sciences Institute, Pearse Street
       
All welcome and admission is free
Details: http://www.biochemistry.tcd.ie/news/publiclectures.php

Tuesday, December 3, 2013

3/12/2013: Some top level results from PISA 2012: Part 2

In previous post I covered the PISA results (top-level overview) for Ireland (http://trueeconomics.blogspot.ie/2013/12/3122013-some-top-level-results-from.html). Here's a summary table showing 2012 data and changes on previous survey:





  • As you can see, Ireland is ranked in a respectable 20th Mathematics place based on the mean score. 
  • We rank 29th in terms of top performers in mathematics and 17th in terms of share of bottom performers in mathematics. This suggests that our system is performing better in raising students from the bottom than promoting students to the top. The first is a good thing, the latter is a bad thing.
  • Another bad thing is that we are losing scores, down 0.6 points annually on previous survey.
  • Ireland's maths score is statistically indistinguishable from the scores for Viet Nam, Austria, Australia, Slovenia, Denmark, New Zealand, Czech Republic, France, United Kingdom
  • And we are statistically above the OECD average along with 22 other countries.

Relationship between annualised change in performance and average PISA 2003 mathematics scores:

  • In reading, our mean score of 523 ranks us between 7th and 9th in the world - highly respectable achievement. However, the score is down 0.9 points year on year.
  • We are comfortably ahead of OECD average here
  • Our score on reading is statistically indistinguishable from Finland, Chinese Taipei, Canada, Poland, Liechtenstein.
Relationship between annualised change in performance and average PISA 2000 reading scores

  • In science we have a score of 522 which ranks us 14th to 15th which is good. Even better: the score has increased by 2.2 points year on year.
  • We are statistically above the OECD average and our science score is indistinguishable from that of Viet Nam, Poland, Canada, Liechtenstein, Germany, Chinese Taipei, Netherlands, Australia, Macao-China, New Zealand, Switzerland, United Kingdom.
Relationship between annualised change in science performance and average PISA 2006 science scores

3/12/2013: Some top level results from PISA 2012: Irish Education Performance

PISA results are out today. Here's the link: http://www.oecd.org/pisa/keyfindings/pisa-2012-results.htm

And some snapshots of Ireland's performance:



And comparatives to Hong Kong, Korea and Shanghai (just because we think we have the best educated workforce in the world):
Note: Ireland = Blue
Hong Kong:
 Shanghai:
Korea:

And to some European peers:
Finland:

Germany:

Sweden:

And so on. Not take these too seriously as standardised testing is hardly indicative of many important aspects and traits relating to education, but... we are reasonably good in reading skills, mediocre in mathematics and sciences... Improving in sciences, flat in maths, slightly down in reading... oh, and our top students are not really at the races, while our bottom students are. So who do we compete against? The brightest or the lowest performers? Makes you wonder, right?

Sunday, April 28, 2013

28/4/2013: A must-view TED talk

I rarely post on TED talks for a reason - aiming high, they often deliver flat repackaging of the known - but this one is worth listening to:

http://www.collective-evolution.com/2013/04/10/banned-ted-talk-rupert-sheldrake-the-science-delusion/

It has been a long running topic of many conversations I have had over the years with some of you, always taking place in private discussions, rather than in public media, that modern science is a belief-based system. My position on this stems not from a dogmatic view of science, but from a simple philosophical realisation that all sciences are based on axiomatic bases for subsequent inquiry. As axioms are by definition non-provable concepts, then the very scientific method itself is limited in its applications by the bounds of these axioms.

This is not invalidate scientific method or sciences, but to put some humbleness into occasionally arrogant position held by many (especially non-scientists) that elevates science above arts, religions, beliefs, and other systems of understanding or narrating the reality.

Saturday, October 6, 2012

6/10/2012: Art, Mathematics and Science


And to finish off the night of charts galore, here's something charts-graphs related of a softer variety: on art and maths . Enjoy!