The fill came mostly from Needham

The firm of Goss and Munson extended railroad lines to quarries in Needham, Massachusetts, 9 miles (14 km) away; a 35-car train carrying gravel and other fill arrived every 45 minutes, day and night.[11] When the Needham gravel pits were exhausted, additional fill was found in Canton, Dedham, Hyde Park, and Westwood.[12] William Dean Howells recalled "the beginnings of Commonwealth Avenue, and the other streets of the Back Bay, laid out with their basements left hollowed in the made land, which the gravel trains were yet making out of the westward hills."[13]

https://en.m.wikipedia.org/wiki/Back_Bay,_Boston