Cultivated cassava diversity has been shaped and is influenced by its wild relatives gene pools. Its primary gene pool (GP-1) consists of the flabellifolia-peruviana subspecies complex that cross easily with domesticated plants. Its secondary gene pool (GP-2) spans Manihot wild species that have a limited ability to interbreed with cassava. RAD sequencing of 640 accessions from the core collection at CIAT, with increased sampling of GP-1 and GP-2, shows a kinship connectivity of 9% and 175 nonrelated individuals. GP-2 sources of cultivated material include but are not limited M. glaziovii. A clear subdivision of the flabellifolia complex is not congruent with the previously postulated peruviana-flabellifolia-tristis taxonomic division. At least 5 different genetically isolated populations around the Amazon Basin could be founding stocks of current cultivated Manihot esculenta. Phylogenetic and pedigree analysis show a hierarchical structure of the populations in Latin America and the Caribbean in both wild and cultivated cassava with extensive gene flow and admixture. This hierarchical structure is dominated by kinship relations, with 95% of the sequenced samples having a third degree relative, and 75 % belonging to a single 1st degree cluster. We propose that this pedigree structure most likely is the result of modern breeding programs carried out by institutions like CIAT and EMBRAPA. Nevertheless strong phylogenetic signal can be detected from the unrelated accession set that could go back to the columbian exchange and holocene domestication. Consequences on breeding and the study of the domestication process are further discussed.