Is it our job to judge unbelievers outside the church?
No. Scripture teaches that those outside the church are God's to judge; judging the unsaved world is His prerogative, never ours. Believers are instead to deal with sin inside the church, restoring in meekness, bearing burdens, and removing a wicked person from their fellowship if unrepentant.
Paul writes that those who are without, the unsaved world, God judgeth, and that is His prerogative and never ours. So judging unbelievers outside the church is not the believer's job. The context contrasts this with the church environment: Paul says to put away from among yourselves that wicked person after having dealt with them, trying to get them to recognize their sin and confess it and get victory over it. If they refuse and will not, then Paul says put them aside until they come to their senses. Earlier, the instruction was to go to such a person in a spirit of meekness, not pride, realizing we also could be tempted and fall into the same trap. We are to bear one another's burdens, help them overcome weaknesses, and encourage them that there is no need to be a Christian failure or constantly live in sin. Thus, judgment of outsiders belongs to God, while church discipline and restoration are the responsibility inside the fellowship.
But them that are without God judgeth. Therefore put away from among yourselves that wicked person.