Psalm 32:1-2 "Blessed is he whose transgression is forgiven, whose sin is covered. Blessed is the man to whom the Lord does not impute iniquity..." Psalm 103:12 "As far as the east is from the west, So far has He removed our transgressions from us". A more clear definition also can be found in Isaiah 43:25, "I, even I, am He who blots out your transgressions for My own sake; And I will not remember sins."
It appears the New Testament draws on the verses in the Old Testament that shows where God forgives sins [since the transgression is against him, as David says, "Against you and you alone have I sinned" (Psalm 51:4) only he can be the one to forgive the transgression]. In the passage about the paralytic in Mark, chapter 2:7 shows how in the understanding of the Jews it is only God who can forgive sins "Why does this man speak blasphemies like this? Who can forgive sins but God alone?"
Hope that helps a bit. I imagine others can contribute more verses.