Well, I've not tried learning many languages - only French, Japanese, Russian and Latvian, and those last 2 I haven't done much of. I am still very much a beginner in Japanese and Latvian, but out of the two, I find Latvian much harder to get an immediate grasp of, because it uses 7 cases and so there is quite a lot of grammar learning right at the beginning. Japanese of course has the difficulty of kanji, but that isn't quite as hard as I expected. The grammar is also easier for a beginner - but I expect that difficulty to increase as I start learning more, because no grammar is really 'easy'!
French I didn't find really difficult at all, but I was learning that since the age of 5 (maybe a bit younger) and also the alphabet is Western so... not the kanji or cyrillic script difficulty.