"Too deep to clear them away..."
See in text (Text of the Poem)
Moss, unlike ivy, doesn't take root in its substrate or upset a foundation's structural integrity. Instead, it anchors itself by way of single-celled rhizoids that cling to surfaces. Clearing it away wouldn't be a terribly difficult process, but would take some time, especially if the moss has grown as thick as the speaker suggests.