Was Paul quoting rumors about himself in 2 Corinthians 10?

Yes, in 2 Corinthians 10 Paul quotes rumors about himself that had come back to him. The context shows he addresses reports concerning his letters and authority, noting that he heard what was said. He defends his Lord-given authority as intended for edification, not destruction.

Les explains that in 2 Corinthians 10 Paul is quoting rumors that had come back to him, and he notes that Paul heard just about everything that was ever said about him. The particular rumor concerned his letters; they were making reference to his first letter, where he corrected abuses and called a spade a spade. Paul says he does not want to seem as if he would terrify them with letters. Les also points out Paul's authority was given by the Lord for edification, not destruction—a flashback to his former days as a persecutor.

2 Corinthians 10:9

That I may not seem as if I would terrify you by letters.

Where did the accusation that Paul terrifies people with his letters come from?

It originated from rumors that circulated back to Paul. Those rumors pointed to his first letter, where he corrected the Corinthians' abuses sharply, called a spade a spade, and made no bones about it, leaving them feeling he was harsh and terrifying with his letters.