It's indeed much easier to list what I like about Go, but here goes:
1) The "carrot dangling": No matter how much you study and play, there is always the feeling that you're "so close" to "getting" another aspect of the game, that you're on the doorsteps of a greater understanding, just to learn very quickly that even you somewhat grasp a new concept, your overall knowledge is of the game is still very limited. Differently worded, it's not enjoyable to pretty much always chase after something you know you'll never attain. And even if you "get" some pieces, putting it all together is an entirely different issue.
2) The time it takes to get "really good" at the game. With most other games (especially newer ones), you can reach a level of competence within a few months of dedication, and you can "compete"; you won't feel hopelessly behind. In Go, unless you started learning when you were six, or at least fairly early in your life (and then ideally in a nurturing environment), you can't ever hope to be "good". This is particularly frustrating if you start learning in the second half of your 30s, as I did. Similarly, seeing teens picking up the game and progressing to dan level within literally a few months inevitably causes me to feel "hopeless" and "too old".
3) Go fosters this very feeling of being "hopeless", "dumb" and "too old" way too often in me. On an intellectual level, I realise that such feelings are silly and that they hint at deeper issues, e.g. partially low self-esteem, wrong priorities, too much comparing to what others do while only focusing on their strengths rather than considering that there are many things I'm pretty darn good at (but naturally I hinge it all on Go), and all of this could be worked on, but is it worth the effort? Therapy might be cheaper and faster!
Seriously, though, Go seems to inevitably uncover and expose one's shortcomings and I wonder if there is really much benefit in it. There are numerous other games I could play that still teach me new "things" without making me feel useless and "too damn old" half the time I spend time with them. From a mental health perspective, getting decent at ten or twenty different games is likely to be more beneficial than becoming an expert at one specific game. At the brink of 40, do I really need more things in my life that make me feel old? The discovery of individual grey hairs and the back pain when getting up on rainy mornings already does plenty enough! Why not engage in activities that produce more moments of success than moments of failure and despair?
4) The time investment: Go takes up too much time if you're not blessed with "talent", "young age" or are genuinely satisfied with your current level of ability. Playing games is fine, reviewing them in earnest is already a time sink. Studying is too, and while I occasionally like book study and have way more Go books than I care to admit, I have to honestly say that most of them are completely and utterly boring and mind-numbing. This is especially true for all these 1970 and 1980 works that are all likely to put me to sleep (with just a couple of exceptions). It almost feels like physical labour to get through some of them, and it feels unnatural and difficult to feed my mind with them. I almost always feel strong resistance when trying to go through them. I learn much better from less dull (and less dense) textbooks, but unfortunately very few of those exist in the Go world. Books can be fun. Go books usually aren't anything but.
5) I don't like the competitive aspect at all. I know all the rationalizations about how Go is a game of two partners, and that winning or losing doesn't matter. But honestly? I think it's bull. Sure, I delude myself with those thoughts too, but at the end of the day you have a winner and a loser, and if you're the loser, you can tell yourself as much as you want that you're the "actual winner" because you learned more than your victorious opponent, but you still lost the battle and making up reasons why you "didn't really lose" feels like ego stroking and sugar-coating more than anything. I don't like competing with others in my recreation time. I'm much happier to compete with myself, and that's why I prefer doing tsumego. Or other solitary activities, like replaying pro games. And psuedo and actual social stuff, like kibitzing, hanging out here on the forums, etc. It's odd, but as much as I love many aspects of Go, I don't particularly enjoy playing the game. The time/enjoyment ratio with some iPad/iPhone games is much higher than my time/enjoyment ratio when it comes to Go, and my brain benefits just the same, (This whole paragraph nay be related with the "fear of feeling dumb" mentioned above, however.)
And now, now I am wondering why I actually play Go, and if this isn't just a really unproductive, frustrating and self-harming way of spending one's time if taking it beyond a very casual, "just for fun" level.
. I think the nervousness would go away if I played more, but until recently I haven't had the time (and the nervousness doesn't help 
). My competitive streak probably contributes to alot of my nervousness because I want to play well, but never feel that I do.
). The game seems to be constantly evolving, and its pretty awesome there is always something new to learn. And to be honest, its nice to have a game that requires a good deal of thought, generally has a very friendly community, and tends to show me my own faults quite often 