Is it my job to judge unbelievers outside the church?
No, judging unbelievers outside the church is not your responsibility. Scripture states that God judges those who are without; that is His prerogative and never ours. The church's role is to address sin within its own fellowship, not to condemn the unsaved world.
Les teaches from 1 Corinthians 5 that Paul makes a clear distinction between those outside the church and those within. Speaking of “those who are without the unsaved world,” Paul says, “God judgeth.” Les emphasizes that judgment of unbelievers is God’s prerogative and never ours. The church is not called to judge the unsaved. Instead, Paul addresses the church environment: when a wicked person is among you, deal with them, try to get them to recognize their sin, confess it, and get victory over it. If they refuse, then put them aside until they come to their senses. So the believer’s responsibility is internal restoration, not external condemnation of unbelievers.
But them that are without God judgeth. Therefore put away from among yourselves that wicked person.