Wow... How did you manage to preserve so much diversity ? It makes me feel simply envious. I wish it had been the same here in France. Occitan dialects have been virtually erased by centralized, compulsory education. Only very old people in rural areas speak some occitan nowadays, and most of them imperfectly. I deplore it. My grandparents' language had a flavor of its own. It was as harsh as the lives they lived, and its mere phonology, in itself, "told a story".
Breton and Alsatian have been similarly affected, though to a lesser degree. Basque seems to fare slightly better, owing to strong connections with the larger Basque area in Spain. But the overall trend is towards uniformity. Which to me means impoverishment.
That's the trend even in Italy, despite the fact that the government recognizes a lot of "languages". It began with Mussolini, who like all nationalists believed it's easier to have a unified strong country if everyone speaks the same language. (That's why nationalists from all around Italy, beginning in the Middle Ages, pushed to have Tuscan as the "Italian" language.) He is the one who mandated that "Tuscan Italian" be taught in all the schools. The next step was the extremely large internal migration from south to north, strongest in areas in Northwest Italy in the beginning, where industrialization first took place. In order to speak to one another people turned to "standard" Tuscan Italian. One of the biggest influences, imo, was television. In order to reach the broadest audience, it was all in "standard" Italian.
In addition to geographical differences, there was also a "class" element. I don't speak the dialects of my own region, and even rather imperfectly understand them, because my father forbade anyone to speak it in front of me. He even tried to correct my pronunciation so it didn't sound so "local". It was quite a "thing" with him. When we came to America he would have me "perform" (recite poetry or pieces of prose) to show off my perfect, "pure" Italian.
He thought, as you say, that the dialects were harsh, and uneducated, and in addition he was a fervent nationalist of the Mazzini school.
Anyway, the map is misleading in some ways. In the south, people still speak a lot of dialect. It's a balancing act: with people they don't know they start out with standard Italian, they also speak it in formal settings, and again, the more educated the person the more likely they are not to speak the dialect. In the Veneto, with its long continuity as a separate kingdom, they much more often speak their dialect among themselves, even if they're relatively educated. In my own area, the dialects are virtually gone. There's been a recent push to bring them back a bit, with street signs appearing in both "languages", and a few hours a week devoted to dialect in school, but imo it's not coming back: too much immigration. It's also just plain ineffficient for communicating. In the last few years a very popular series was done about Neapolitan organized crime called "Gomorrah". It had to be broadcast with Italian subtitles.
In tv series like Montalbano, they have only supporting, minor characters speak in Sicilian dialects, and even then it's Italian influenced, or it wouldn't have the audience they wanted.
Anyway, as to why so much diversity in language? Same reason(s) there's so much diversity in food and customs. Partly it's geography, partly it's different political systems for so many hundreds of years, but I think partly we just don't like change, and we're inordinately attached to our own local "ways". It's called "campanilismo": attachment to and identification to only the area within sound of our own church bells.
It has both negatives and positives, like everything else in life, and I'm "guilty" of it too, if not in regard to language.
