Aluminium flashing into gal gutter and your gutter will be rusted out in short time by electrolysis so no go.
Two issues to fix:
1. Leaking gutter - So long as the gutter is cleaned according to instructions silicone sealants for gal should last 20-30 years. But this is probably best repaired using a full two-part waterproofing membrane system - talk to Bunnings. There are a number of product that will do the job.
2. Overflow - this can be fixed by effectively extending the gutter higher up the rafters on each side using ponding boards or sheets. These are lengths of galvanised flat iron bent to have a lip that sits down into the box gutter 30-50 mm and sit on the rafters under the tiles or sheeting - on top of battens usually.
They are pop riveted to the existing gutter, but if access is an issue can even just sit in place with silicon sealant liberally applied underneath to seal between the gutter and the back of the ponding sheet. If tiled roof then lift tiles and take the sheet say 300 mm up the rafters - a big increase in the holding capacity of the box gutter. If colourbond or similar roofing then simply undo the screws sufficiently to slide the ponding sheet up and under.
The other issue is the size of the outlet - it won't matter how big the box gutter is or is extended by ponding boards to be if the outlet is a 90 mm or 100 mm downpipe into a 100 mm storm water drain. So many houses have overflowing gutters because they have several downpipes (and/or a box gutter) all of which flow into a single 100 mm stormwater connection.
As others have said you need to enlarge the outlet at the end of the box gutter to the max and also have at least one overflow grate/ sump at ground level so that in downpours there is somewhere to ease the restriction of flow. And as was also said - check there are no blockages in the stormwater system.