First of all, did you shoot it with the bayonet extended? They are sighted in that way, and will shoot off POA if it's folded back.
As for drifting the sight, they were factory sighted and tight so the soldier in the field wouldn't screw around with it. The best description of drifting the front sight is that you need two people, one to hold it in place on the bench, the otehr beats a BIG brass drift with a big hammer, and it'll move a little. Shoot, repeat as necessary or until you're too tired.