The power conversion efficiencies (PCEs) of organic solar cells (OSCs) have now reached high values approaching 20%. These recent advances in OSCs based on non-fullerene acceptors (NFAs) come with reduced non-radiative voltage losses. In contrast to the energy-gap-law dependence observed in conventional donor:fullerene blends, the non-radiative voltage losses in state-of-the-art donor:NFA organic solar cells show no correlation with the energies of charge-transfer electronic states at donor:acceptor interfaces. By combining dynamic vibronic simulations with temperature-dependent electroluminescence experiments, we provide a unified description of non-radiative voltage losses for both fullerene- and NFA-based devices. We highlight that the photoluminescence yield of the pristine materials defines the lower limit of non-radiative voltage losses. We further investigate the dominant non-radiative recombination processes in state-of-the-art NFAs, providing rational materials design rules for enhancing the PCEs of OSCs by increasing the photoluminescence yield of NFAs.