FST and kinship for arbitrary population structures I: Generalized definitions

Abstract

Abstract F ST is a fundamental measure of genetic differentiation and population structure, currently defined for subdivided populations. F ST in practice typically assumes independent, non-overlapping subpopulations , which all split simultaneously from their last common ancestral population so that genetic drift in each subpopulation is probabilistically independent of the other subpopulations. We introduce a generalized F ST definition for arbitrary population structures, where individuals may be related in arbitrary ways, allowing for arbitrary probabilistic dependence among individuals. Our definitions are built on identity-by-descent (IBD) probabilities that relate individuals through inbreeding and kinship coefficients. We generalize F ST as the mean inbreeding coefficient of the individuals’ local populations relative to their last common ancestral population. We show that the generalized definition agrees with Wright’s original and the independent subpopulation definitions as special cases. We define a novel coancestry model based on “individual-specific allele frequencies” and prove that its parameters correspond to probabilistic kinship coefficients. Lastly, we extend the Pritchard-Stephens-Donnelly admixture model in the context of our coancestry model and calculate its F ST . To motivate this work, we include a summary of analyses we have carried out in follow-up papers, where our new approach has been applied to simulations and global human data, showcasing the complexity of human population structure, demonstrating our success in estimating kinship and F ST , and the shortcomings of existing approaches. The probabilistic framework we introduce here provides a theoretical foundation that extends F ST in terms of inbreeding and kinship coefficients to arbitrary population structures, paving the way for new estimators and novel analyses.