Can a building permit be denied for potential illegal use?

Can a building permit be denied for potential illegal use?

While reviewing plans for a building permit, code and zoning officials often raise objections citing potential illegal use due to proposed work.  Is that justified?