I used to work with retards, but I just called them people. Or clients, or kids, or even "those f'ing kids who won't shut up," or whatever. Some of my favorite people in this world are retards. Generally they're nicer than non-retards. But I don't think it's okay to use "retard" as a slur. As long as you use the word, my clients go through life being treated as less than people, because they're just retards, and as you said, retards are "shit that doesn't matter."