Removing Vision would solve the problem; Therefore youd still need scouting or the use of another scouting effect.
It only gives a small vision buff to flagpoles and it doesn't target units in the FoW. It mostly relies on either friendly units to spot or sight from the flag poles.
However I figured out that the flag pole sight buff from Anvil Tactics stacks with the flag pole sight buff from PO and can make most of friendly territory visible in smaller maps because flagpoles get like 40 vision in total when PO is active with Anvil researched. Which is a pretty big problem to be honest as it does essentially make the ability mostly self-spotting.