There's a lot of cultural stigma attached to small penises. They're not considered attractive. We mock guys who aren't well-endowed and consider them less desirable than guys who are more blessed. A lot of a guy's perceived masculinity is tied up in the size of his junk, which is just utterly absurd.
Once upon a time, I was a very sheltered girl. Sex was the ultimate taboo - I thought it was dangerous, dirty, and I failed to understand how anyone could derive pleasure from it. It wasn't until I got to college, and started meeting more experienced, sex-positive, empowered ladies that I started to cultivate a more positive view of sexuality.
Much to the dismay of my male readers, I have to admit that size does in fact matter to us ladies. Sure, it isn't the only component of good sex, but it definitely contributes. After all, the motion of the ocean doesn't matter much if you can't feel the waves!