Drawing lots as a tie-breaker

The peer review process and discussion by experts are the key methods used by the SNSF to assess the quality of research projects. However, it is not always possible to decide with a simple yes or no whether an application merits financial support. The majority of the submitted applications are of similar quality - with some outliers below and above. However, not all of the applications in the upper middle range of the rankings can be funded because of the limited budget; the evaluators are tasked with making a selection. To isolate the most convincing projects, they define a "funding line" that divides the field into fundable and non-fundable projects, depending on whether they are above or below the line.

Drawing of lots to prevent subconscious biases

In rare cases the quality of applications cannot be differentiated any further; in effect, they notch up the exact same score. Such cases involve the risk of funding decisions being influenced by the subconscious biases of evaluators, e.g. with regard to background, interdisciplinarity or even the readiness to take risks. This is evidenced by research done into evaluation processes. These situations show us the limits of peer review and evaluation by expert bodies, which can result in supposedly objective funding decisions that are in reality tainted by bias. Such outcomes can be largely prevented by drawing lots.

Implementation following successful pilot phase

The SNSF used a lottery procedure during a pilot phase for the Postdoc.Mobility career funding scheme between 2018 and 2020. Following an in-depth evaluation, the Presiding Board of the National Research Council decided to allow evaluators to draw lots if necessary in all of its funding schemes as of 2021. "Only very few applications will be subject to the drawing of lots," says Matthias Egger, the President of the Research Council. "This method is reserved for proposals that are very close to the funding line."

But isn't there the risk of a random decision being unjust or even blocking someone's career? According to Matthias Egger, the opposite is true: "The process will only be applied to projects that are equally good from a scientific point of view. In such cases drawing lots is the fairest solution because it is blind and rules out bias."

There are no reservations from a legal point of view either. In 2011, the Federal Supreme Court set out the requirements for randomised decisions. The procedure must be transparent and credible. Lots may be drawn either physically or electronically, provided there is a level playing field for all contestants.

Pieces of paper from a capsule

The SNSF will implement a physical drawing of lots. If during an evaluation a situation arises in which two or more applications cannot be differentiated any further based on objective criteria, their numbers will be written on pieces of paper. Each piece of paper will be inserted into a non-transparent capsule. A member of the body or of the SNSF Administrative Offices will then draw the capsules from a clear bowl one by one. The resulting ranking - based purely on chance - finally spells out which applications lie above the funding line and which below it. The entire process is documented and recorded on film.

If the fate of an application was decided by drawing lots, this is explicitly mentioned in the approval or rejection letter sent to the applicants. The transparency of the procedure is thereby increased. Moreover, researchers who are not awarded a grant at least learn that their application was of very high quality even if it could ultimately not be funded. The pilot phase has shown that such decisions are on the whole accepted.

In the future, the method of drawing lots may be used in all SNSF funding schemes. It continues to be part of the selection process for Postdoc.Mobility and may be used in the Sinergia evaluation process with immediate effect. Also in project funding, lots were drawn as a tiebreaker for equally highly rated proposals in mathematics, the natural and the engineering sciences for the first time in March 2021. "Only 9 of the 278 evaluated proposals were affected, a small share," says Matthias Egger. "But it made sense to draw lots specifically in these few cases."

Lots also drawn in other countries

The SNSF is not alone in applying this method. New Zealand introduced the drawing of lots already in 2015 in a national funding scheme. Since 2017 Germany's largest private funder, the Volkswagen Foundation in Hannover, has relied on randomised decisions for the allocation of a small number of grants. And FWF, the Austrian public research funder, is using this method in its 1000 ideas programme.

"By allowing for the drawing of lots in all our funding schemes, we are forging ahead with the development of our evaluation processes," says Matthias Egger. "In cases that are too close to call, this randomised process makes selection fairer for the researchers involved and utimately benefits all of our grant applicants."

• Organisational Regulations of the National Research Council, Article 23 paragraph 5 (PDF)
• Matthias Egger on the drawing of lots