It definitely takes the fun out of a potential conversation with someone who you thought shared the same interests as you.

I remember hearing someone say that if you're going to wear a t-shirt of (a band, political statement, musician etc.) then you better be prepared to talk about it or at least have some knowledge about it.

I think it's fine if you know of the band or person and their work but if you have no idea who or what is on your t-shirt than it could be a little awkward.

I remember when Thrasher was popular in high school. Everybody wore Thrasher yet didn't really know anything about it. It just looked cool and everyone was wearing it.