I'm not so sure. It seems a little too broad.

My point is, who says they were drinking it, and not fermenting it or changing it into cheese at that point?

Other than that, it's standard selection for a beneficial trait among herding people.

"The scientists examined eight skeletons excavated in Sudan and Kenya, which were between 2000 and 6000 years old. They scraped hardened dental calculus from their teeth and looked for known milk-specific proteins trapped inside.The findings revealed these people were consuming some sort of dairy product at least 6000 years ago, the team reports today in Nature Communications. That makes this the earliest known direct evidence for dairy consumption in Africa, and perhaps the world.
The research also shows dairying in Africa goes back just as far as it does in Europe—perhaps longer. That undercuts a myth, propagated by white supremacists, that lactase persistence and milk drinking are somehow associated with white Europeans.
What’s more, ancient Africans don’t appear to have evolved any milk digesting genes, according to a study of some of their skeletal DNA published in 2020. “It looks like the community was drinking milk before they had lactase persistence,” says Madeleine Bleasdale, a co-author of the new work and a specialist in ancient proteins at the Max Planck Institute for the Science of Human History."