Apsrt from reload times this is already how it generally works.
But you also should not forget that TDs often function a bit differently in factions. For USF the TD is their onlyreal AT unit, UKF only needs the Firefly against the heaviest targets because they already have the Comet, and SOV are somewhat in between but usually also need an SU85 (in team games at least).
Axis factions however have a dual setup with a 'light' TD version and the heavier Panther. Giving both the same treatment does destroy unit variety.
You're right, but IMO all factions only need 3 AT options - tank destroyer, AT gun, infantry AT. Infantry AT should counter tank destroyer, AT gun should priority counter medium and than heavy, tank destroyer should counter priority counter heavy/medium.
You're right in that unit variety would change, but I don't think it would have to be sacrificed. For example, you could have Axis AT guns fire slower but have higher penetration, making them stronger against heavies and weaker against mediums (the Rak 43/Panzershreck actually had really high penetration). Allied AT guns could have less penetration but higher fire rate. In order to compensate for heavily-armored Axis tanks, they could toggle between standard AP and APDS/HVAP; APDS and HVAP slow down the fire rate of the AT gun, and could cost a small amount of munitions per shot if necessary. Similarly, for tank destroyers you'd be able to differentiate between light TDs and heavy TDs. StuG III, Achilles, and Wolverine fall under this category. If I had developed the game, I would have nerfed the SU85 penetration and made it a light TD, while using the SU100 as a non-doctrinal heavy TD with good frontal armor, extremely high peneration (~211 mm at 500 m with APBC), and fast speed, but slow reload rate like the Firefly. Since SU-76 doesn't do a good job of filling the gap between AT power, either SU85 would be made a stock light TD or T34/85 would be a stock upgrade option on the T34, giving a solid counter to medium tanks (after all, Soviets only have 14 units compared to 15 of the other factions). USF wouldn't really need a major change; IRL the Jackson had better fire rate than the Firefly and SU-100 (but worse penetration, not that it really matters on anything smaller than a King Tiger), and was actually pretty slow (26 mph). If need be, the M18 or M10 could have been made a stock light TD.
Anyway, these are all just ideas that I've been mulling over for some time. Such radical changes aren't needed in the current state of the game, although maybe they could prove useful in a mod or a future CoH, if one ever comes.