We ask FT participants for feedback at a couple of points in the process; first when they are notified whether they got time, and again a few months later when they are hopefully writing up their excellent results. The initial survey contains a mixture of multiple choice questions, and boxes in which people can give their own answers, comments, and suggestions. Here’s a snapshot of the results from that survey so far:
Most people have reviewed telescope proposals before, and a similar number find the reviewing process about as easy or difficult as they had expected. The words most frequently used to describe the process are “interesting” and “educational”. Admittedly, this is probably prompted by the fact that we ask “How did you find the reviewing process? E.g. interesting, challenging, educational, time-consuming”, but people do seem to agree that they benefit from reviewing others’ proposals.
We also ask how the quality of the FT proposals compares to those they have reviewed in other contexts. The large majority of participants rate the FT proposals as similar to others they have assessed, with a couple even finding the FT proposals of higher quality. One or two reservations were expressed in the comments. For example:
“Quality was comparable to HST proposals for the most part, though there were clearly some (~30%) that were hastily put together.”
Users of the FT program receive all the reviews written about their proposals, and we have always hoped that this feedback would be helpful. This does seem to be the case, with >75% of users reporting that the comments on their proposals were “mostly helpful”. As one user commented:
“The practice of providing each proposer with the full set of reviewer reports (rather than a diluted summary) was unusual, and I welcome the change. This provides much deeper feedback.”
The respondents aren’t very bothered about the possibility of the peer review system being abused. Only one person reported being “moderately concerned”. Of course, this is a biased sample; if you’re worried about the review system you’re probably not going to submit a proposal in the first place. Nonetheless, it is gratifying that, having been through the whole process, these people generally saw no major cause for concern. Almost everyone said they’d be “quite likely” or “very likely” to use the FT program in the future.
The feedback surveys have also given us a number of suggestions for improvement. Some of these (e.g. expanding to Gemini South, doing away with separate FT observing nights) have already been implemented, some (e.g. shortening the proposal form vs not shortening it) don’t agree with each other, and others (e.g. “the final decision tree is not visible to the reviewers”) are on our to-do list. If you use the program in the future, please be aware that we do appreciate these surveys and we are paying attention to the results.