...and again the question - 'Can I blame the Y on his ... (whatever)?'
The size of the testicles: Gorillas have harems, no sexual competition and there balls are really small. Chimpanzees have extreme competition and theirs are 10 time bigger than those from the gorillas (compared to their body mass). Those of humans, by the way, are right in the middle of the two extremes. So, does the Y determine the size? Obiously it's the social circumstances and not the genes.
To determine what the Y-chromosome can and cannot, we should know what's on it. There are 4 kinds of genes there.:
1. The pseudoautosomal genes (regions PAR1,2,3), which lie on positions, where the X and the Y can pair and segregate. They are therefore not male specific, so we can forget them.
2. X-degenerate genes: those are genes which come from the precursor of the X and Y chromosome. Most of them are already deleted, some exist as defunct pseudogenes and just a few of them (a dozen or so) are still functional. They are usually cell housekeeping genes with no pronounced functions which we can relate to ANY 'visible' attributes of a person, but a few of them indeed have special functions. Just a handful of them is currently known. AMELY is related to the enamel production, NGLN4Y binds to the nerve cell synapses and can be blamed for several neural dysfunctions, TBL1Y is involved in cell signalling processes and finally SRY, THE male-defining gene, which is highly variable, except for a strictly conserved motif, which is necessary for sex definition. This gene initiates the forming of the male gonads. Mutations can lead to reduced size of the testes, but also to complete sex reversal. It interacts with the Androgen receptor, which can cause androgen insensitivity and therefore turn an otherwise normal male into a person with reduced male expression, both somatically and psychologically.
3. Amplicon genes: those are housekeeping genes for germ cell development transposed from the autosomal gene pool to the Y chromosome. There they were duplicated or multiplied (therefore - amplicons) to enhance or make their abilities more specifically. In reality they are less than a dozen genes, but with amplification there are about 60 of them around, accounting for about 80% of the protein coding genes. Their effect is almost exclusively on the sperm cells themselves and have no somatic consequences.
4. X-transposed genes: When God created Adam about 4 Mio years ago
, he shuffled some genes from the X-chromosome to the Y. From the two, which have survived, one - TGIF2L - is a transcription factor, a housekeeping gene, that is targeted mainly at the testes (perhaps at the brain and the prostate as well). The other gene is PCDH11Y, which is one of the important genes for the development of brain lateralisation, so it is involved in the mental sex difference of male and female, and plays a role in the handedness of a person.
That's all about the functional genes on the Y genome, hardly more than 20 different genes. As you can see, there are less than a handful of genes, which can be responsible for physical appearance, and the mutations, defining the different haplogroups, have to be exactly inside these genes, and the mutations must have the capacity to change the molecules, they encode, sufficently for a recognizable variation in a person.