"Mass scales as the cube of radius. So if the Earth's radius is 4000 miles, then a 100-mile wide (50 mi radius) asteroid only increases Earth's mass by 0.0002% Or 1.4% for a body the size of Texas -- which would be larger than what formed any of the near-side basins on the Moon, and maybe larger than what formed Hellas on Mars. Only a few such large bodies struck after the Moon, Mars, or Mercury formed their crusts, which the Moon at least did near 4.5 billion years ago. "

If the Earth is adding .0002% to it's mass each year, in a million years one has added 200% to this 4000 mile radius Earth's mass. But as you say 100 mile wide asteriods are rare and the larger asteriods have far more mass than the smaller impacts but the smaller impactors occur more commonly.
Also one keep in mind the with these very very large impactors they can throw into orbit or escape a large percentage of their mass- especial with smaller planets [but this ejected matter from Earth size object in mostly going to say closer to Earth orbit and probably impact the Earth again. Whereas with size of "impactors" which landing on Earth currently, daily, weekly, monthly, yearly this mass is not being ejected from planet with an atmosphere.]

Going back to the 100 mile diameter rock, let's say it's density is 1 and change it to 160 km rock [sphere = (4/3) pi r cubed] gives 1,206,370 cubic km and each cubic km weighs a billion tons or 1.2 e18 kg.
Whereas Earth has a mass of: 5.972e24 kg.
So, as I said it is unlikely that Earth has added 1.2e18 kg mass on an
average yearly rate. Or a accretion rate that create a Earth's mass in about 5 million years.
Ok, some loose numbers:

Say there is 1 km radius rock per 100 years [density 1, mass: 2.356 billion tons] or average of 23.56 million tons per year [2.356 e10 kg].
Today's rate could said to be about than .1 km radius rock per 100 year
[density 1, mass: 2.356 Million tons] or 23,560 tons per year*. [So we talking about a thousand fold increase is the rate as we have currently and btw this would radically change our life if this were occuring- on a yearly basis cities would be destroyed, global climate would altered, and coastlines would be killing fields.]
A 1 km radius rock per 100 years means that .1 km radius rocks impactor far more frequent than 100 years, it also means that 5-10 km radius rock [dinasour killer rocks] occur in the thousand of years timeframe instead of the millions of years timeframe. Instead of dinosaur extinction event occurring every 100 million say it's 10,000- 100,000 years. As far as adding mass to Earth we can roughly assume that most of the mass would not leave a Earth size body, nor create a moon.
5 km radius rock [density 1, mass: 294.5 billion tons] or average of 29.45 million tons per year [per 10,000 year event].
10 km radius rock [density 1 mass 2,356 billion tons or average of 23.56 million tons per year [per 100,000 year event].

I could keep doing this for 20, 40, 80, and 160 km radius, OR: 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160 km radius. OR:15, 20, 25, 30,... ect Km radius. The larger rocks are far more massive but they also far more rarer events.

*When I say .1 km per 100 year I'm try use a loose measure which includes the .1 size but also the "spectrum" of 10 km size [100 of millions year time period] through to cosmic dust which rains down hourly, so even though .1 km radius don't occur every 100 years, the total average amount is probably more than 23,560 tons per year [if it was less than 1/2 this amount I would be surprised, if it was 10 times more this amount I wouldn't be surprised].

So, if we use the 1 km radius per Century measure. Just the 1 km impactors would add around 2.356 e10 kg per year, but if we dealing with period say 500 million years we have include all the material from the dust all the way to a 160 km monster and this could mean instead of 2.356 e10 kg per year, it could easily be 10-100 times this amount: 2.356 e11 to e12 kg per year. giving 1.178 E20 to 21 kg over this 500 million period. This is obviously a small amount of mass considering the Moon is 7.35e22 kg. And the rate would probably need to be more than this, but it's probably between 1-10 km radius impactor per Century and if nearer to 10 km then their could mean more of the impactors which are 160 km size range and also requiring far less time than 500 million years.