If a block fails validation early (e.g. if it is complete nonsense) then nothing will be sent down to detail validators, and they will not generate a 'Go'. Consensus should be prepared for this, and any 'NoGo' should stop waiting for 'Go's.
This also means it will need to synchronise more carefully (which is safer design anyway), and ignore any subsequent 'Go's that do arrive after the NoGo. This requires match on the block hash, since numbers will be reused after a rollback.