I would say that overpowering a boat is not illegal. If the boat was marked for a max horsepower of 150 hp and you mount a 170 hp motor, but use less than the 170hp, then all should be fine.

They do that with aircraft all the time, where engine operation in a specific airframe is limited by RPM which limits the hp output of that engine.

If the engine is too heavy for the transom, that's a different story.

Dave