What NOT to do
NEVER punish your cat when it is in the litter box. Even if the cat is urinating or defecating over the edge of the box, it is still in the box and that behavior needs to be rewarded. The box is either too small, too shallow, or the cat cannot squat sufficiently low if it eliminates over the edge. In essence, the failure lies with the resources being given to the cat, not the cat's behavior itself.
NEVER trap your cat in the litter box to give it medicine, brush it's teeth, or take it to the vet or other car ride. The litter box will then be associated with fear of unpleasant events rather than comfort and relief of bodily needs.
NEVER strike your cat or inflict other painful punishment.
Cats are not small dogs!
Behavioral Modification
People often think that they can train a cat like they saw their parents or grandparents train their dog growing up. However, cats do not respond the same way to training as dogs do (and methods for training dogs have changed in recent years as well!)
Dogs are pack animals and their behavior reflects level of submission to those in a higher social position in the pack. Cats are more solitary in their interactions and although social hierarchies can and do exist in multiple cat settings, the cats do not have the automatic submissive behavior with punishment that dogs have.
Cats respond best to positive reinforcement to modify their behavior. This means that you positively reward with attention or food the behavior you want to have continue.
Every time you see the cat in the litter box, even if it is only digging or scratching, praise it. When cats that are not very social with each other show tolerant or indifferent behavior, praise them.
We want the cat to associate using the litter box with good things like praise and food treats.