This is kind of weird in the middle of the sentence. It really interrupts the flow. And who said it? You can get around this by paraphrasing. How about this as an alternative: "Fewer than 3,000 Southerners owned more than 100 slaves in 1860, yet the pro-slavery . . ."