Frequently Asked Questions about the material in the Methodology Guide to the NRC Assessment of Doctoral Programs

1. Why is the NRC presenting ratings and rankings at all? Aren’t the data sufficient on their own?
   The data alone are extremely useful, but we have been asked to do this. We think that we can improve on our own past work as well as the work of the commercial world. Programs and institutions really wished to have some overall view of program quality, and the NRC had been asked repeatedly if it would update the study completed in 1995. After a preliminary report Assessing Research Doctorate Programs:  A Methodology Study issued in 2003 it was decided that we could respond to this desire and address some of the perceived weaknesses for the prior NRC reports including “spurious precision”.
   
   
2. What is the basis for the criteria adopted to assess graduate programs?
   They come from the PhD programs themselves. In each academic field we asked a large proportion of faculty members to tell us what characteristics they believed were important to have in a strong program. We also found the characteristics that were common to the programs that the assessors felt to be strong, and then we combined these two independent measures to give an overall set of criteria.
   
   
3. Why will the rankings be given in ranges instead of as a single number?
   The committee that is conducting the study strongly feels that a single number for a rating would be misleading since the input data inevitably reflect statistical fluctuations and the formulae that we use to put weights on the data also contain uncertainties. All of the numbers that went into the assessments contained statistical uncertainties, and we felt that it was important that these uncertainties be reflected in a transparent fashion.
   
   
4. What do the ranges mean?
   When we queried different faculty members we received different answers for the most important metrics, and, if we used different years for the data, we obtained different input data, so to calculate one rating for one program we would pick particular values for each case. We repeated this selection 500 times to calculate a range of values for each program. If half of the values were between “A” and “B” then we said that the range of the rating for that program was “A to B” as our best answer to presenting the results with their real uncertainties.
   
   
5. Were all of the “weights” given to program characteristics the same for each field?
   The statistically calculated weights are the same for all the programs in a field for one run of the model.  They may vary from run to run because the values of the weights from different subsets of the surveyed faculty vary.  We conducted 500 runs and show the weights for the middle 50% of these runs.
   
   
6. Why was each program’s ranking calculated 500 times?
   The rankings for all the programs in a particular field were calculated 500 times in order to reflect, as much as possible, all the sources of uncertainty.  These are:  variation across assessors, year-to-year variation in the measures, and variation due to the statistical estimation.  500 isn’t a magic number.  It seemed large enough to reflect adequately all these sources of variation.
    
7. When will the final rankings be released?
   With the publication of the Methodology Guide, we have completed a major milestone. The NRC and its committee feel an enormous obligation to produce data and rankings that are of the highest quality possible.  Striving for this objective requires careful review of the data and rankings for over 5000 programs.  This has already taken considerable time and is nearing completion.  The committee must also finish summarizing the most important findings from the data, as well as discuss the strengths and shortcomings of the methodology for incorporation  in the final report.  The last step of the process is for the final report to undergo the Academy’s rigorous review.  We are working to complete all this work as expeditiously as possible.
   
    
8. How do you count books for the humanities?  Do you count edited books?  Do books only count if they are from a major publisher?
   We did not count edited books, but every non-edited book received a weight of 5, compared to a scholarly article, which received a weight of 1.  We counted all books and did not take into account who published them.
   
    
9. On page 19, you list 3 tables institutional coordinators will receive.  These relate to the overall rankings.  Will you also be distributing ranking ranges for the dimensional measures?
   Institutional coordinators will receive the ranking ranges for the dimensional measures, as well as for the overall measures.