Previous studies identified 3 Neolithic Han Chinese super-grandfather Y haplotypes, O2a2b1a1a-F5, O2a2b1a2a1-F46, and O2a1b1a1a1a-F11, but their relationships with the archaeological and written records remain unexplored. We here report genome wide DNA data for 12 ancient samples (0.02x-1.28x) from China ranging from 6500 to 2500 years before present (YBP). They belonged to 4 different genetic groups, designated as Dashanqian (DSQ) of Xiajiadian Culture in the Northeast, Banpo (BP) of middle Yangshao Culture in the Central West, Zhengzhou Xishan (ZX) of Miaodigou Culture in the Central Plains, and Others. Present day F5 samples were closer in autosomal distances to the ZX and DSQ groups while F11, C, O1, and O2 samples were closer to the BP group. We also sequenced the Y chromosome of one of these ancient samples K12 from DSQ and found both K12 and a previously reported ~4000 year old sample MG48 from Northwest China to have the O2a2b1a1a1a2a-F2137 haplotype, belonging to the most prolific branch O2a2b1a1a1-F438 immediately under F5. We further found close relationships between ZX and DSQ and between ZX and ancient M117 Tibetans or present day Southwest Dai Chinese carrying the F5 subtype O2a2b1a1a6, implicating radiations of F5 subtypes from the putative place of F5 origin in ZX. These results are remarkably consistent with archaeological and written records.