Xrhrft

A (homebrew) magic greatsword has this feature:

Heavy Blow. When you take the Attack action on your turn, you may choose to forfeit your attacks for one very powerful attack. The powerful attack uses the lowest modifier in the attacks you forfeit. If you hit with this very powerful attack, you deal damage as if you had hit every attack you forfeit, and [...]

The intended behavior is that an 11th level fighter with a +8 to hit would be able to use his action to make a single attack with a +8 to hit, dealing 6d6+12 damage on a hit (2d6*3+STR*3).

If this fighter has great weapon master, then the attack would be a +3 to hit to deal 6d6+32 damage on a hit (2d6*3+STR*3+10*3).

How would an effect like this be implemented to most accurately follow the design conventions of 5e?

The key phrase is "extra attack", and you need a couple other changes

It's "Extra Attack" that you're talking about:

Extra Attack

Beginning at 5th level, you can attack twice, instead of once, whenever you take the Attack action on your turn.

The number of attacks increases to three when you reach 11th level in this class and to four when you reach 20th level in this class.

And you wouldn't "forfeit" extra attack in your Attack action. You would use an Action to activate the Heavy Blow feature of your magic item (great sword), and say something like:

Heavy Blow

You can use an action to make one melee attack which, on a hit, does damage equal to the total damage you would make if you hit with this weapon using all attacks in your Extra Attack feature (if you have that feature).

The reasons I think this follows the writing style convention better is (1) magic items in 5e commonly say that "You can use an action to..." do something whereas (2) there is not really a model that I know of in 5e for a magic item having you "forfeit" your extra attacks.

This appears to be an item designed for 3rd edition of D&D where the attack bonus and number of secondary attacks depended on your primary attack's base attack bonus. Under that system, you would have the choice to give up your best attack for a chance to do double damage on your second best attack, or give up your two best attacks for a chance to do triple damage on your third best attack and so on.

Multiple attacks in 5th edition don't work that way at all so there is no easy or direct port.

The best way to port these is to capture the essence of the item and look at things that do that sort of thing. Like the second point of Great Weapon Master:

Before you make a melee attack with a heavy weapon that you are proficient with, you can choose to take a -5 penalty to the attack roll. If the attack hits, you add +10 to the attack's damage.

You could use this directly once you decide how it interacts with somebody who actually has this feat.

You do not "forfeit attacks" in 5e. Your original phrasing leaves a lot of wiggle room for overpowered combinations, and needs to specify what benefits can apply as if it's a melee attack and what benefits do not, and needs to specify that this is not an attack action. Try this phrasing;

"While wielding this greatsword, you may use your action to deliver a heavy blow with the greatsword. Make a greatsword melee attack. If it hits, calculate damage as if you used your action to attack with a greatsword the maximum times you are able as an attack action.

You may apply modifiers from class features, racial features, spell effects, and character feats to the initial attack roll and subsequent damage calculations as if making a melee weapon attack, but only if they could reasonably apply to all attacks on your turn. Limited use features like divine smite, single-opportunity advantage, or true strike cannot be applied to a heavy blow action."