Background: The grading evaluation of atlas based auto-segmentation (ABAS) of organs at risk (OARs) in thorax was studied. Materials and Methods: Forty patients with thoracic cancer were included in this study, and for each thirteen thoracic OARs were delineated by an experienced radiation oncologist. The patients were randomly grouped into the training and the test dataset (20 each). The investigated ABAS strategies included single-atlas (Single), majority voting with 5 atlas matches (MV5) and simultaneous truth and performance level estimation (STAPLE) with 5 atlas matches (ST5). The Dice similarity coefficient (DSC), the difference of the Euclidean distance between centers of mass (ΔCMD), the difference of volume (ΔV), maximum Hausdorff distance (MHD) and average Hausdorff distance (AHD) between auto-segmented and manual contours were calculated. Results: Most of the index values (33/65) of ST5 were optimal. There were differences in the grading results for the five indexes. With DSC, five, four and four OARs were graded into Level 3, Level 2 and Level 1, respectively. The mean DSC values ranged from 0.88 to 0.96, from 0.73 to 0.79, and from 0.53 to 0.62 for the Level 3, Level 2 and Level 1, respectively. Conclusion: Grading evaluation of ABAS of thoracic OARs based on the DSC proved to be feasible and relatively more reliable. The thoracic OARs auto-segmentation was divided into three levels based on the DSC. Level 3 OARs can be auto-segmented, Level 2 OARs delineations need to be manually modified after the auto-segmentation, and Level 1 OARs are not recommended for the auto-segmentation.