@Maleth,
I hope I didn't mislead you; while there is definitely Berber ancestry present in North African Jews, it is a minor part of their ancestry.
This graphic is still pretty accurate and might help in visualizing the relationships:
The Jews, as a maritime commercial community like the Phoenicians before them for a rough analogy, had diaspora communities before the destruction of the Temple and their scattering by the Romans, although unlike the Phoenicians they retained their distinctive identity. There were Jews in North Africa and Iberia, for example, long before then, and some degree of admixture took place. (Of course, in the Roman Era refugees would have swelled their ranks.) Some, if not most, of the North African Jews probably then moved into Iberia with the Arab conquest, and with the expulsions, some of them then moved back to North Africa, bearing some Iberian input. This is why they now follow the Sephardic (Spanish) rite in their religious practices. It's a complicated story. What is clear, however, is that modern North African Jews do not have the up to 25% SSA input that is present in many North Africans today. (The North African Jewish communities are also amazingly distinct from one another.)
You may in fact be familiar with some North African Jews without being aware of it.
The philosophers Bernard Henri Levy and Jacques Derrida, for example:
Jacques Derrida:
Claude Cohen Tannoudji-Nobel prize winning physicist:
Gregory Fitoussi-actor. He was recently the lead in the series "Selfridges".