That's consistent with Sam's station as a humble gardener, not a member of the loose aristocracy class like Frodo or Merry and Pippin, who had the leisure and wont to go out a-walking just for fun - and also those latter had relatives near the south of the Shore, and so would have reason to visit the area.