You would definitely end up with some metal debris passing into the coolant galleries. How much grief or risk of damage it could potentially cause? - don't know. It could jam the thermostat, get lodged in the heater valve etc. I would remove the manifold. It isn't necessary to tap and block the coolant port as it was standard practice just to seal over it with the gasket and adapter, but it massively reduce the risk of a coolant leak.