Stalling in the Safari Zone adds to your own experience. You don't expect to see your Pokemon instantly in the Zone. Yes, we're all guilty of making the Zone a bit to excessive in the RP element, and that may need to be calmed down a bit, but there is no true intentional stalling for the Zone,
As for the length of Double Battles vs Single Battles. the problem is that that is how they were introduced-in both the Anime and Games and being equal. The concept of double battle concept came about as the new norm, despite it being a dual on one one basically.
Since its a new norm, it starts at the base level.
The ref isn't penalized. As Elf said, it is due to the ref choosing to accept this match. They have to be ready for anything, whether its fast or slow. You have the ability to choose your matches, and if you end up in a match that is not to your liking, you also have the option to bow out and get a sub-ref.