New Science's NIH report: highlights

New Science just published an excellent report on the NIH, both a good primer for those curious about how the world's premier science funding institution works as well as an essay packed with insights that go beyond the obvious. A recent theme of my latest few essays is the key importance of tacit knowledge in many contexts, and in science reform in particular. I noted elsewhere that the claim "the HHMI funds better researchers than NIH in aggregate" was generally believed to be true before Azoulay (2011) came out¹ . Accordingly, the essay was written not by consulting public sources, but by unearthing common knowledge within the NIH world by directly talking to researchers that work or have worked within the system. Relatedly, another interesting read, making complementary points, is Reforming Peer Review at NIH from the recently launched Good Science Project.

I agree with the overall point of the essay: The NIH is at the moment an invaluable and irreplaceable institution. It can and should be improved, and there is room for a more diverse funding ecosystem which would be preferrable to the current situation where a single institution, to a large extent, orchestrates the world's life sciences research.

Here I wanted to single out specific points the essay makes that I found particularly interesting:

The single most consistent criticism of the NIH that I heard from sources, across all issues, was that the organization is too “conservative.” That is, too conservative in an institutional sense, not an ideological sense.

The NIH is considered insufficiently willing to take risks. This can be seen in its consensus-based grant evaluation, the de facto discouragement of ambitious grants, its drift away from basic research, and the lopsided distribution of grants which favor large, established organizations and researchers.

This is something that one can read in many places in the internet (that the NIH is conservative in its funding). We can tentatively take this as true based on relatively general agreement. An interesting next step, not suggested in the essay, is asking scientists for specific kinds of research that wouldn't get funded. Perhaps brief examples of grants and a brief rationale for why that research would probably not get funded despite its putative merits. Collecting those would help understand in a more fine-grained way exactly what sort of biases the NIH process has. Workshops with senior scientists to discuss NIH grants and what should be funded or not could be held with traditional and new funders to help bridge that gap in understanding. Open Philantrophy could have released the grants they chose from th Transformative Research Award but chose not to, probablyd due to confidentiality reasons. Making those grants accessible

One interviewee related stories of two instances when their grants were rejected because they involved technology not ongoing in their lab and, thus, there was no preliminary data. In the latter case, the interviewee had been receiving NIH grants for over forty years, they had served as an editor on a major journal, and had been an advisor for an NIH institute. All that clout and history wasn’t enough to get the grant approved. While such ability to withstand political forces is impressive, the reason the grant wasn’t approved was that the interviewee never worked in the field in which they applied for the grant.

Fortunately, the proposal later caught the attention of a prominent non-profit. The interviewee submitted a one-page application and they “nearly fell off [their] chair” a few months later when they got full approval at a higher funding amount than expected. Their project has since yielded “transformational” progress in the field, and though the interviewee is extremely positive about the NIH overall, they are concerned about the lack of risk-tolerance in study sections.

I almost thought: Ed Boyden! But no, Ed Boyden is too young to have been applying for grants for forty years. But this matches his story with expansion microscopy.

A few researchers had an interesting take on an unintended consequence of this system: the NIH is biased against “super nerds.” Navigating the “benevolent ponzi scheme” requires anticipating the judgments of colleagues, knowing the right people to talk to for advice, plotting out how to stagger grant timing and explain results that diverge from official grant applications. These are all skills correlated with extraversion, networking, and sociability. They are not the typical traits of a socially awkward scientist who loves to spend hours going through data sets and discussing abstract theories, rather than figuring out how to game complex bureaucratic systems.

This is not to say that a researcher can’t be both a great scientist and a skilled player of the game. But there are certainly researchers who are uncomfortable with the system, and who wish they could spend more of their time on the science and less on figuring out how to get to do the science.

This was also the case back in the day in the Rockefeller Foundation in the 40-50s. Though I could not retrieve the document ("N.S. Notes on Officer's Techniques"), there is a section in there on this and I have the relevant quote here: I know of a man who is almost surely the best expert on the genetics of oak trees in the world; but he doesn't take baths, and he swears so much and so violently that most persons just won't work with him. As a result he is living a frustrated and defeated life; and he not only doesn't have any students - he doesn't even have a job. Weaver then goes on to say that they would't fund people like this because team work is key in science. This may be an extreme case, but it is an example against the thought that "anti- super-nerd" bias is a recent phenomenon.

On the other hand, there are some researchers who are probably a bit too comfortable with complexities embedded in the grant system. Whether by design or happenstance, some lab leaders gain reputations at being so good at getting grants that they focus most of their energy on getting resources and then leave the actual science to their staff. Then again, maybe a bunch of super nerds working for a master grant-getter is the ideal lab structure?

This, along with other points (the redistribution that occurs with indirect costs, the benevolent Ponzi scheme, etc) is an interesting example of why things may not be as bad as they seem. Maybe they are not efficient, but the system as a whole seems to have found workarounds for issues that would otherwise make the research enterprise more inefficient than it actually is.

The NIH gives about 50% of all extramural grant money to 2% of applying organizations, most of which are universities or research facilities attached to universities. 38 The top 10 NIH recipients (out of 2,632 institutions) 39 received $6.5 billion in 2020. This is 22% of the NIH’s total, extramural grant budget ($29.5 billion40), and 16% of the NIH’s entire budget.

In 2020, the top ten largest recipients of NIH money were: 41

1. Johns Hopkins University - $807 million through 1,452 awards
2. Fred Hutchinson Cancer Research Center - $758 million through 301 awards
3. University of California San Francisco - $686 million through 1,388 awards
4. University of California Los Angeles - $673 million through 884 awards
5. University of Michigan Ann Arbor - $642 million through 1,326 awards
6. Duke University - $607 million through 931 awards
7. University of Pennsylvania - $594 million through 1,267 awards
8. University of Pittsburgh at Pittsburgh - $570 million through 1,158 awards
9. Stanford University - $561 million through 1,084 awards
10. Columbia University Health Sciences - $559 million through 1,003 awards

Here it would have been interesting to look at per-researcher figures as well, with the caveat the indirect costs and intra-university redistribution would not be accounter for if looking at the PI level. But nonetheless, what kind of PI gets more funding, and for what. Is it mostly driven by lab size? Would be interesting to know!

The essay also discussed the Grant Support Index. I made some remarks on that here and especially here:

A few years ago, there was a debate around whether NIH should cap funding for individual researchers, on the grounds that there are decreasing marginal returns to concentrating funding on a single investigator. Opponents argued that such a policy would unfairly penalize successful investigators leading large labs that are doing highly impactful work. [6] The proposal was ultimately scrapped. It’s not relevant whether such a proposal would have worked: both sides had reasonable arguments. What is important is that at no point did NIH think of randomizing or trialing this policy at a smaller scale—they designed it from the outset as a policy to affect the entirety of NIH’s budget.

That is the kind of thinking that we need to change. Instead, NIH should have considered selecting a subset of investigators and applying a cap to them, and then compared results a decade into the future with those that were left to accumulate more traditional funding.

I am now less bullish on RCTing our way through science reform but I do still think that enabling experimentation at smaller scales would a good step forward.

On researcher age I wrote an essay just on that question where I end up thinking that at the end of the day, relative to the current situation, a younger workforce would be good, hence my idea of a Young Researchers Research Institute.

On study sections, there are videos on youtube that show how they work in practice. I found these very helpful when I first came across the "study section" concept.

In sum, it can take 14 months before a grant submission results in funding. Meanwhile, existing grants will be running out. Postdocs will be coming and going from the lab. New discoveries will pull research in one direction while other research paths peter out. PIs will have to take all of this complex management and budgeting into account while going through a process which typically takes 6-14 months to pay out.

This is unacceptable and while true that not every grants program needs to aspire to be Fast Grants, cutting down time by at least half seems within the bounds of possibility. The gains in the responsivity of the system are most likely worth whatever minor accuracy in review quality could occur. The essay has a good sampling of ways to reform the current study section model.

The NIH’s extramural grant system is somewhat convoluted, but it has a logic. It is designed to promote private research with government money while simultaneously fostering research institutions through subsidies.

When the NIH awards extramural grants to researchers, the funds are divided between direct costs and indirect costs.

Direct costs are funds given to the researcher. These funds cover costs that are easily associated with the researcher and their team, including salaries, supplies, equipment, and lab space.

Part 5 has a good explanation of how indirect costs work and why they are there, and presents the pro and con case for high indirects. I tend to side with the "they are too high" position and while universities may complain they don't have enough money, being forced to tighten their belts could force efficiencies: perhaps more investments in reducing costs in sequencing, imaging, or pathology cores, making them more like autonomous business units that charge researchers the cost of what they want to do, and extra to account for maintenance and new purchases. Somewhat less realistically, scientists could be encouraged to shop around for shared facilities at other research establishments to induce some healthy competition. A move towards transparent pricing and less cross-subsidies seems on net good. Here I think: Could we do a case study for a particular university, and see how their indirects work? The incentives might not be there. But the NIH could commission a survey of indirect utilizations, including interviews with adminstrators and scientists, to see exactly what is happening with those indirects, and what the implications of reducing them might be.