Taken from Bulbapedia:
When you throw a Poké Ball at a wild Pokémon, the game uses a formula based on the wild Pokémon's current health, any status effect it may have, and that Pokémon's Catch Rate, to determine your chances of catching that Pokémon. The formula is as follows:


Chance = floor (((( TotalHP * 4 ) - ( CurrentHP * 2 )) * CatchRate ) / TotalHP ) + Status + 1

Note: floor means to make the number an integer by rounding downwards, i.e. even floor(1.999) would be equal to 1.

CurrentHP is either the wild Pokémon's current HP, or if its current HP is greater than 255, use floor ( CurrentHP / 4).

TotalHP works exactly like CurrentHP, except using the Pokémon's maximum HP value.

CatchRate uses the wild Pokémon's Catch Rate, found below.

Status - If the wild Pokémon is asleep or frozen, Status = 10. If it is paralyzed, poisoned, or burnt, Status = 5. Otherwise, Status = 0.


The chance is further modified according to the kind of Poké Ball you use; for a full list see Poké Ball. After all these values have been calculated, the game computes a random number anywhere from 0 to 255 inclusive. If this random number is less than the Chance calculated above, the Pokémon is caught.

---

EDIT: Groudon and Kyogre's catch rate is 5. If you use Great Ball, the chance is boosted 1.5 times. If you use Ultra Ball, it's 2 times as effective.