Paul uses the words "not to company" with certain who call themselves brethren, that might be better in this case because he distinguishes between those of this world (without) and those who are within (who is called a brother) but who might be these things.
1Cr 5
9 I wrote unto you in an epistle not to company with fornicators:
10 Yet not altogether with the fornicators of this world, or with the covetous, or extortioners, or with idolaters; for then must ye needs go out of the world.
11But now I have written unto you not to keep company, if any man that is called a brother be a fornicator, or covetous, or an idolater, or a railer, or a drunkard, or an extortioner; with such an one no not to eat.
Paul says these were to be put away from them
1 Cr 5
12 For what have I to do to judge them also that are without? do not ye judge them that are within?
13 But them that are without God judgeth. Therefore put away from among yourselves that wicked person.
Now here (the next letter to the Corinthians) Paul also says
2 Cr
[11] O ye Corinthians, our mouth is open unto you, our heart is enlarged.
[12] Ye are not straitened in us, but ye are straitened in your own bowels.
[13] Now for a recompence in the same, (I speak as unto my children,) be ye also enlarged.
[14] Be ye not unequally yoked together with unbelievers: for what fellowship hath righteousness with unrighteousness? and what communion hath light with darkness?
[15] And what concord hath Christ with Belial? or what part hath he that believeth with an infidel?
[16] And what agreement hath the temple of God with idols? for ye are the temple of the living God; as God hath said, I will dwell in them, and walk in them; and I will be their God, and they shall be my people.
[17] Wherefore come out from among them, and be ye separate, saith the Lord, and touch not the unclean thing; and I will receive you,
[18] And will be a Father unto you, and ye shall be my sons and daughters, saith the Lord Almighty.
