You have to think of the lesser of two evils. Either you keep them here and either try and spend money assimilating them to your culture so that they don't cause trouble or spend money cleaning up after them (e.g. through extra police, emergency services etc.), or you spend that money deporting them, only for a portion of them to come back over the border again.
Alternatively, drawing inspiration from kainashi's post, make it so committing a crime while being an illegal immigrant a capital crime, or send them to Gitmo or something where they have no rights. It would certainly deter illegals from committing crime.