Well, think of it this way -- would you rather be tackled by a Goldeen or a Machamp? Sure, the attack's power is the same, but the attacker definitely isn't. I'm not saying the modifier should be big (maybe +2 or +3 damage depending on the size and speed of the attacker versus the size of the defender), but it should be a little more apparent than it is right now.

I'm also talking to myself about this too; I'm very much guilty of using pre-calculated damage when it comes to reffing.