All the posts in this blog have been about the input side of the proposal writing process and are based on long experience of what inputs lead to the best outputs i.e., what and how to write to get funded. The evaluation process is the filter between the optimistic and expectant researchers and the fate that awaits them some months down the line. And it would be a great idea to find out as much as possible about this critical phase of the process. Of course we can read the ERC information about the steps and we can know the names of the panels. But, as ever, the really important information is the tacit knowledge gained from having done the process and learnt about the culture of the institution in which it takes place. This is the kind of information that helps turn the black box into something a bit more transparent.
Luckily there are some reports back from evaluators involved in the process. I came across a very good one recently and thought I’d spend a post looking at how my sense of what is a good proposal corresponds or diverges from the opinions of an experienced evaluator. The presentation I’ll refer to is by Prof. Sašo Džeroski from the Jozef Stefan Institute, Ljubljana, Slovenia who is a very experienced researcher, project leader and evaluator and was involved in the ERC evaluation of the StGs in 2014 for panel PE6, The presentation is called ‘ERC Grant Applications: a look from the other side’ and can be found at http://tinyurl.com/go73ycc.
There are many fascinating insights here from all parts of the evaluation process. For example on the topic the number of reviews we learn that in each of the two phases the proposal is reviewed by three members of the panel. In phase 1, two members of previous panels reviewed the proposal remotely and so each proposal was reviewed five times. In phase two external reviewers suggested by panel members are invited to review and a proposal may be reviewed by between two and ten external reviewers in phase 2, so a total of between five and thirteen reviews in total. This is quite a large variation between the least and most read proposals and it is not really clear what this information means on its own despite being a curious fact about the variation within the evaluation process. I think we can take from it that the proposal had better be easy to read! and the presentation covers this ground in later slides as we will see.
Some very interesting facts emerge about the panel member workload. In total this panel received 214 proposals and the author of this presentation personally got 42 to review in phase 1 and 10 in phase 2. A noteworthy fact is that in phase 1 the ERC pays panel members for half an hour for each proposal and this includes both reading the proposal and writing the review – we should keep this in mind when writing as some evaluators will be trying to keep to this time allocation for various reasons e.g., if their pile is even bigger! The author of this presentation states that he, in practice, spent an hour reading and then more time for review writing and while we can guess that this kind of generosity and scrupulousness might be fairly common among reviewers we can’t take it for granted and assume that it will be like that across the board. My own reading of many review comments, for example, suggests that it is certainly not universal. The headline from this slide is the last point that the writer makes – they won’t spend much time on it: this I am convinced is something that researchers writing for these calls needs to keep in the forefront of their thinking.
The next slide deals with an issue that I think is really critical and has been the topic of various earlier posts on how the writing has to be phased carefully to appeal to a general reader before the technical details are explored more fully in the later sections of the work. The author here states very clearly that the proposal must be written in a way that can be understood by the ‘generalist’ i.e., someone from the broader area of your work but who is not a specialist in the particular topic where you focus. The point is made very strongly indeed here, and I quote ‘I CANNOT EMPHASIZE ENOUGH THE IMPORTANCE OF THIS’ (slide 11). The author thinks that if the proposal is ‘too specialist’ it is likely that the comment coming back will reflect this i.e., that the proposal is not written to appeal to the generalist despite its having potential.
While personally I haven’t seen a comment of this nature I bow to the author’s knowledge in this respect and certainly agree wholeheartedly with the idea that the work needs to be pitched for a generalist. I would tend to say, I think, that this is more important in the first phases of the work where the argument should be engaging and draw the reader in giving the generalist enough support to see the importance of the specialist points that can be made as the work progresses. Overall, I think the guiding principle should be to make it as open and as transparent as possible, avoiding jargon and supporting the reader with plenty of sign posts and guidance as you head towards the more technical areas of the work. It might be the case that specialist knowledge will be unavoidable in the methods section for example but this is not a problem if the objectives (which will win and lose the job here) are open enough to the well informed generalist reader and they have been expertly guided through any complexity in the opening phases of the work.
Other excellent points are made throughout the slides and new insights and knowledge is too plentiful to do justice to here. I’ll just mention in passing a comment about the interviews in phase 2 on slide 15: ‘The interviews MATTER, can change rank drastically’. The demands on researchers are really high here, every aspect of the work and every aspect of its presentation are critical to the final outcome and the panels and searching very actively for the best all-round candidates. The candidate can’t let their guard down for a moment – hardly surprising when the competition is so tough and the pay-off in terms of financial and reputation benefits are so significant.
Jumping down (to slide 17) to the section on what makes a good proposal from the evaluators’ side is interesting as it is corresponds closely to the main points of the recommendations I have made in the blog posts over the last year. It is encouraging to see that my view and the view of an evaluator are in harmony on so many important issues. The first point is to ensure that it ‘Is clearly written’, well yes, this is beyond dispute despite being rather rare. ‘Is on a topic considered to be important (by peers)’, this it seems to me is true. I would even go as far as to say that if your first thoughts and plans for an ERC proposal are not in a topic that you know is considered to be currently important for peers then you should consider shifting focus until your track record and a hot topic map over as far as possible. I think you need to take a very pragmatic and flexible approach to finding the ‘sweet spot’ where what is feasible matches up as closely as possible to what is most talked about in the field and commonly understood as important. I think the process of finding the topic to propose has to be done a bit more actively and with a focus on winning than it often is – most researchers take the next step in the field they are ploughing away at and if it is attractive they are lucky and if not the carry on regardless in most cases which I don’t think is the most competitive behaviour to adopt.
The next point the presentation makes is that it is important that the work ‘Identifies major gaps in knowledge about the topic’ and again I am pleased to say that this has been a major point made in the training sessions, project reviews and in this blog as I consider it to be a very important part of a good proposal. The state-of-the-art section of the proposal needs to be constructed very carefully to highlight the gaps in knowledge, and the bigger and more critical the gaps are (as long as they are ‘project sized’ rather than, for example ‘programme sized’) the better and easier it is to make the case for the proposal above its competitors. Identifying the gaps in the first phases of the argument is a very important early step in the flow of the argument – in the early phases it is in many ways about telling a story of knowledge failures and near misses in the state-of-the-art and building your project around them as this presentation goes on to say in the next point.
A good project it claims is one that ‘Sets objectives that would bridge these gaps’ and this is, I think is very well stated and correct – the objectives are located right in the gaps in the state-of-the-art and are designed to take the field forward by overcoming the hurdles and barriers in the field: the simplest and best way to generate objectives is to highlight gaps in the state-of-the-art and then demonstrate that you’ll close them and how you’ll do this. To set the objectives in these gaps in knowledge ensures that they are precise and relevant and right up to date and so is the quickest way to make them attractive to the evaluator.
The presentation also makes some interesting points about the kinds of impact that the project should be promising and which are in line with some of the early posts in this blog about who the target audience is for the ERC programmes and projects. It is nice, we read on page 18 to have outputs that are relevant to industry and society but these impacts do not have to be direct or short term. I think this is an interesting point to make and one that I make as clearly as possible in the training sessions I do for those thinking of writing a proposal. The ERC is unique in that it offers the researcher a respite from having to produce useful outputs that can be taken up by stakeholders in the wider community whether this be by civil society or government stakeholders or industrial ones. As the presentation goes on to say ‘Theoretical research with impact on the development of its field stands a good chance’ – this point I always also emphasise. The target audience is first and foremost peers in your field and opening up new horizons and pointing towards new frontiers for research is the primary task of an ERC project. It is a unique chance for scientists to speak directly to other scientists at a theoretical or basic science level and this is a fundamental part of the ‘spirit’ of the ERC and what sets it apart from the range of other calls under H2020. Therefore, I think that it is a very important idea to keep in mind and well worth emphasising.
The last point I’ll look at here from this presentation concerns how the proposal must set itself apart. Again this is interesting and reassuring as the points made are very much along the same lines as those I make over and over again in the training and the project reviews. It is important, we read, to set the work apart from ‘other research in the field and previous work of PI’ – a good point to keep in mind and it is the job of the state-of-the-art section (again, this is a hard working section!) to show how this is distinct from work elsewhere. The state-of-the-art, as the blog posts on this topic go into in more detail, needs in my view to get down to the level of named researchers and their work and details about the research trajectories and lines already ‘live’ in the field. Also, critically, the proposal also needs to make it clear that while this work is feasible and the researcher credible in the field this not simply a rolling out of work that is ongoing or will be done anyway i.e., it is not incremental work or core work of department that has run out of cash and is insisting that the staff write ERCs to keep the show on the road. It is a balancing trick here, as it is at every stage, to show how this is new but that the track record indicates that it is possible. The Janus-faced nature of the researcher here and elsewhere in the proposal has been covered in earlier blog posts and it is a tricky balance to maintain but a core part of the proposal rhetoric and needs to be considered throughout. As this presentation says, if the word incremental start cropping up in discussion then failure is just around the corner.
There are other points very well worth bearing in mind during the proposal writing phase throughout this excellent presentation which I don’t have space to cover right now as this post has already gone on a bit too long. I will try to find other feedback from evaluators in order to try to find out what they are looking for when they do this job. With information of this sort we can make sure that the proposal is giving them what they are hoping to find, as far as possible. I am very pleased to read that all the points that are considered by the evaluators are covered in the training and review process that we offer at A Bigger Splash Ltd. – I think we are working along the right lines on this. I am also convinced that there are many more things that I have written about in the last year that will give the researcher the edge. However, it has been useful to review things from the other side of the fence and to see that we are on the same page as far as making good ERC proposals is concerned.
A quick word of warning – while all of this is good sense and all of these guidelines mentioned here are sensible to follow there is a tendency sometimes to try to ‘second guess’ or anticipate what the evaluators will think in too much detail. Some researchers can get a little obsessive about wondering what the evaluators are going to think of this or that small detail – I think this is another of the ‘displacement’ behaviours that are pretty common when faced with the stiff challenge of writing for the ERC. It is imperative that wondering about what is going on in the black box doesn’t become an excuse for not doing the writing work. In a nutshell, take these general lessons seriously, keep them in mind and then get to work to write the strongest, most committed, most well locate in the state-of-the-art and most ambitious proposal possible speaking confidently in your own voice at the pressing issues in the field – this ‘authenticity’ will come across and take you much further than trying to read the minds of a dozen reviewers. In the end you need to take the risk, take a stand and write to win, as if a lot depended on it…which it does.