The home side had sailed into a 3-0 half-time lead, but a spirited second-half display meant the Terriers only lost by one goal in the end.
Town's boss said: "We made a poor start and I am bitterly disappointed as they only had three shots in the first half and we were punished three times.
"In the second half we produced the sort of form I know we are capable of and had most of the play - their keeper had to make four outstanding saves.
"At least we showed some character, but we have thrown the points away."