Max Plank, who looked at thermodynamics, and it took him 20 years to reach his conclusions, that, that matter was, was quantized. (Donald Braben)

Max Planck got his PhD around 1879, and it wouldn't be until ~1900 when he published the key papers postulating the quantization of energy, which he used to explain black body radiation (See here for some context). The 1879-1900 gap are probably those 20 years that Braben has in mind. It's interestingly hard to find anything published by Planck prior to this. His 1900 papers don't reference his earlier work. The wikipedia bibliography for Planck starts in 1900. One has to search for a while for this bibliography where we can find (pg. 140 and onwards) that he was indeed publishing papers, including a book, but it seems those papers were not particularly well regarded, most of that work has not even been translated to English [Unlike say, Einstein's Annus Mirabilis papers, or Planck's own 1900 Nobel-winning papers] and it may be a reasonable assumption that they were relatively inconsequential. This would fit with Braben's point: Planck wasn't a stellar academic until his 1900 paper, indeed taking 20 years to achieve an outstanding result. Had Planck been judged on his pre-1900 results he might have been kicked out of the system. Or would he?

thanks to strong backing from Helmholtz—then widely dubbed as the “Reich Chancellor of Science”—on whom Planck’s originality had clearly made a strong impression, he was appointed professor of theoretical physics at the University of Berlin in 1888 at the still remarkably young age of 30. He stayed there until he retired almost half a century later. (Braben)

Whereas Braben frames the story of Planck as the quest of a man for truth that no one could know where it led him to (And that the modern grant system would not have been able to support), I see an alternate story where modern Planck would have been able to do his research just fine, either working on the problem of black body radiation directly, or getting funding for something else (Like his pre-1900 chemistry work) and then working on what he actually wanted to work on anyway on the side, which suboptimal as it is, would have still lead to the same result. Planck did not spend 20 years working on one problem, rather so it happened to take him 20 years to find a problem that would happen to be extremely relevant; at the same time, in the period leading up to that he knew what he could be doing, and could have explained so to a peer review panel.

Funding for longer

The Funding people approach tends to, seemingly, be associated with the idea of funding for longer. The HHMI and NIH director's award get funded for longer than the usual R01-funded NIH investigator.

But there is no necessary connection between the philosophy of choosing someone that you expect will be successful and that of funding that person for long. You could fund people, for a year to do whatever. Or you could fund a project for 20 years without asking for interim reports.

Indeed, Braben's own funding scheme, Venture Research did not seem particularly focused on funding for long, some projects ran for 3 years, others for 10. For the HHMI/NIH cases, the researchers that do get it are more successful, but in my previous post I showed that it's not trivial to disentangle if this is because the researchers are great to begin with or because the award empowers them to do great things. Moreover, even if we knew that it's the latter, we wouldn't know if it is the amount of the award or the prospect of longer term funding that is doing the work here.

Funding for shorter has its advantages: it allows funders to more rapidly redeploy funding to new proposals if needed. But at the same time it can by its nature incentivize a different kind of work that would otherwise be done under longer funding setups. Say Dr. X promises to do Very Important Science, but that it will take him 20 years to do so, and moreover that it will all not seem very exciting up until the end of those 20 years, when it will all make sense and the results obtained will surely earn him the Nobel Prize. Would you fund Dr. X? Maybe not, perhaps if Dr. X insists that he can't be evaluated at all during those 20 years, which induces some suspicion. He's saying that his research plans are not legible, and moreover that there are no intermediate steps that can be used to assess the quality of the final product. This is highly unusual: If Dr. X says he wants to prove that relativity is fake news and that he intends to do so by building a new measurement device and then doing some measurements, the interpretation of which contradicts relativity, the construction of the device can be reported periodically. Prototypes can be shown, and unpromising avenues can be highlighted in periodic reports. If all he wants to do is to do theory, then still intermediate results, and failed approaches can be presented during milestone reviews.

The Human Genome Project took over a decade but produced all sorts of intermediate results. Apollo took somewhat less than a decade from the initial feasibility studies to Apollo 11 but the project was already demoing rockets less than two years after. ARPA projects can run for as little as three years (Azoulay et al., 2019) These are just illustrative examples of a general rule: There are no examples that come to mind of projects that took a long time but that could not be assessed as they progressed.

To me this suggests that the benefits of longer funding (Projects that do need a lot of time to be completed) can be achieved with a string of shorter term milestoned grants. This also has the advantage that research performers (Be it in an academic or industrial context) can be assessed as the project advances, and their funds can be cut partially, completely, renewed, or increased as needs are reassessed each year. It's possible to imagine a model where even with this milestoned approach, a researcher may not propose some piece of work because it's uncertain whether the first reasonable milestone can be delivered within the time allocated. The problem of funding length then becomes the problem of milestone spacing: Maybe the first milestone should be a year or two or three after initial funding, something that is domain dependent.

This problem may be solvable for concrete projects, but is it solvable for funding people? HHMI has set their own funding milestone threshold seven years after becoming an HHMI investigator. HHMI will generally not cut an Investigator a year into the program, but DARPA very well might do so with a project. This is because DARPA will impose specific milestones to be met whereas HHMI will do a broad "Show your work" assessment with just a broad research program proposal upfront that doesn't detail what will be done by when. This sounds reasonable to fund science (Not assessing constantly), but does it have to be seven years? Can we say more about the specifics?

Empirical Evidence?

There isn't really much evidence on whether giving out grants for 1,2, 10 years, or rolling grants if milestones are met or some progress is showed are better than any others.

Manso (2011), and experimentally Ederer & Manso (2012) in the context of fostering innovation in a managerial context, say that the optimal innovation‐motivating incentive scheme exhibits substantial tolerance (or even reward) for early failure and reward for long‐term success. Moreover, commitment to a long‐term compensation plan, job security, and timely feedback on performance are essential to motivate innovation, and shows this using a formal model which is essentially a restatement of my remarks above, but with math. The model does not allow for the possibility of partial disclosure (milestones); that is it considers that a researcher or manager will be judged on the value of the final output solely and not on interim reports on the process, or ongoing results, as would be the case in a real world setting. He is right that for exploratory research one can't just pay for an outcome that is uncertain or totally unknown at the start, but that doesn't mean that the agent can't be assessed through other means. Even when funding the solution to some obscure math problem, and after a year Dr. X only has a bunch of excursions around the problem of interest but not a solution, peers that have done something similar could tell if Dr. X is doing the right thing. Maybe they could even see it as a red flag that he is not failing enough!

Of course, this assumes that one can put some bounds over the uncertainty of the total amount of time required to finish the project. Dr. X may say that 5 years are required for a thorough exploration of the problem, so in that case a midterm assessment after 2.5 years may be ideal.

From the point of view of Dr. X in this example, assuming that privately he knows more or less how long it will take, and in which kind of way he'll be assessed then he does not have an incentive not to do what he wants to do: He'll want to tackle hard problem Y proposing a series of tactics Z to take an estimated time W.

The situation where Dr. X may be deterred is when what he's up to is completely illegible to others. He says —I'm going to do some research around the topic of Y and I need this much money per year

—Ok, can work with that

—But I don't know what I'll find

—Mmmkay, blue skies research and all

—Or when will I finish

—Yeah, we can set you up with a rolling grant, we'll just have some interim evaluations

—But only God can judge me, and I'm an atheist so no one can really, I probably won't have anything to show until completion so you probably won't get a fair view of my progress

—How come?

—I'll just be fumbling around and maybe I'll find an interesting problem to solve on year 1 or on year 10, who knows

—Umm... But we can examine your fumbling and see if you are fumbling in the right way, this is something people have done before, we expect you to fumble, we are funding blue skies research and we know how it works!

—Yeah, you may be right after all

You can see the problem. Even on blue skies research there's something that is being done and can be shown to an external reviewer to show that progress is being made or not. Exploratory research has a structure of its own, and as long as that is acknowledged and not penalized [Just because on year 1 all a PI has to show is a literature review, a failed experiment that disconfirmed a hypothesis, and a bunch of read books doesn't mean their performance is bad, that may have been their plan all along]

The NSF asked their funded PIs about the question. The usual NSF grant is for 3 years and PIs would like an average of 5, closer to the NSF [perhaps they are just looking at the NIH and going for that]. 96% of surveyed PIs think that this increase should lead to an increase ability to pursue high-risk ideas which would make for an interesting test of the hypothesis to see if they are right. Here's what one of them says:

My main concern with NSF is the 3 year cap on funding for grants as I mentioned before This 3 year cap is not enough to have a student finish a project. Even if the project is renewed, then the renewal often does not come in time to support a graduate student’s stipend working on the original project. As a result, the student has to TA or find a scholarship or just borrow from other grants for obtaining money to finish and graduate. Consequently, I would be very much in favor of extending grants to 4 years rather than 3.

Here we see that a key variable to set duration is, empirically, the expected time a project as a matter of fact tends to take, as theorized. When asked [Table IV.3] whether they prefer one of: more money, longer awards, or more awards per year, on average 40% of Pis chooses more money, with 24% on average choosing the longer awards.

Conclusion

There wasn't much to look at! The only thing that can conclude here is that setting aside the cost of doing constant assessments [the costlier, the less frequent], their frequency should increase the easier it is to assess progress [and the more trust the researcher can put in that the reviewers will be fair]. In one extreme, when funding defined projects, one could even do monthly assessments.

Would such a system (Where effectively no one has unconditional funding) have excluded Planck? (The Planck test). I don't think so; Planck would have applied for some early grants to do obscure chemistry research, done that, then progressively apply for grants that get closer to what he ended up doing. His was not a case of an illegible research program that no one could judge, it was a series of progressive steps where he ended up finding an interesting problem, and a solution. It can always be said that he may have failed to secure those grants but the same is true of failing to secure a professorship back then, the problem now is that there are lots of applicants, so given a fixed pot of money and imperfect means of picking up the Plancks of the world, any system is going to end up leaving some unfunded.

There is a case that could be made for grants that run for longer: Less time applying for them! But longer grants, for a fixed pot of money, come at the cost of fewer grants. In turn, that could reduce funding success rates, requiring for more grants to be written in the first place! This may be okay if this causes the quality of each grant to increase because of the following:

Imagine you need to build a new cheaper methylation array that you will then use to develop new methylation clocks. Doing each thing may take 5 years. But the new method is not particularly impressive, it just does the same thing but cheaper, the key advance is using that cost reduction to vastly expand the training set that is then used for the latter new clock idea, which is the one that may get tons of citations. So you could do that or you could do a project that runs for 5 years that is less good but that you are sure can be done in that time. To do the cool project you need to get one grant for five years, and then apply for another one. You might not get the second one, leaving you with work that doesn't look impressive to your colleagues. In view of this you may decide to pursue the project that can show impressive results in 5 years. I see this not as an argument for generalized longer term funding, but for not being rigid with funding terms. In the ideal scenario, the researcher discloses the entire research project upfront and would secure funding for the first part, then be evaluated on the cheaper methylation array and then be guaranteed the next 5 years if that's delivered. This may be a missing grant modality: A research program grant, where a >5 years research project is proposed, broken down into intermediate deliverables. This helps fund projects that are hard to evaluate by peers that are not aware of the overall project's idea, but that can be evaluated by a panel that was initially informed of the original idea. This may be part of the actual secret sauce of HHMI, by considering their Investigators' careers broadly rather than as a string of unrelated grant applications, reviewed by different people each time, they can enable these projects that require longer term funding.

The considerations at the start of this article assume that someone is aware of the research program in question; if that's not the case (And this is the reality for most scientists trying to string together grants) then we are in the "hard to assess in the interim" category of projects which favors longer term assessments. But note that this is not due to a intrinsic feature of the project, but due to the fact that the way the current funding arrangement is set up does not allow for knowledge of a researcher's whole program to be used for their assessment (Unless you are the HHMI or the committee that awards the NIH Director Pioneer's Award or similar).

Suppose that we were the HHMI and were considering a change of the 7 year funding period for 10 or 5 or 4, again with a very generous and risk-tolerant first term review period. What would we do? There is no solid evidence there to decide either way in the general case; HHMI used to give grants for 5 years and then they switched to 7 on the grounds that "what many of these scientists are trying to do – open up new areas of research – is very difficult". That's the only justification that we can find; perhaps they applied a rule of thumb like "If NIH does 5 and we are trying to fund more interesting stuff then 2 more years seems about right". Or maybe they had a very detailed model but I doubt it.

I'd like to know if there's more evidence that could be useful here, so if you have something please send it my way!