It had nothing to do with Xander being Buffy's watcher, that was Giles' job. Basically, Caleb ruined Xander's eye because as Caleb put it Xander sees everything "You're the one who sees everything, aren't you? Well, let's see what we can do about that." Then Caleb sticks his thumb in Xander's eye socket.