Building collapse arising from destructive earthquakes is often the primary cause of casualties and economic loss. Building damage assessment is one of the top priorities in earthquake emergency work. Quad-polarimetric synthetic aperture radar (PolSAR) data not only have the advantages of radar imaging being neither exposed to sunlight nor blocked by clouds, but also contain the most abundant information of the four polarimetric channels. Only using conventional polarimetric decomposition methods may lead to overestimations of the number of collapsed buildings and the exaggeration of the degree of earthquake damage. We proposed a parameter called the sector texture feature of the Fourier amplitude spectrum (STFFAS) to describe frequency-domain texture features based on the Fourier amplitude spectrum in order to solve the overestimation of earthquake building damage. In addition, we proposed a scheme to recognize building earthquake damage using only a single post-earthquake PolSAR image combined with STFFAS and the improved Yamaguchi four-component decomposition method. The 4.14 Ms7.1 Yushu earthquake that occurred in Yushu County, China, in 2010 is taken as the experimental case. Compared with conventional polarimetric decomposition methods, this method successfully separated 70.18% of standing buildings from the ground objects mixed with collapsed buildings, thus significantly improving the extraction accuracy and reliability of building earthquake damage information.