If you're properly invested then holding low interest debt isn't a bad thing - my father, whom I deeply respect, has refinanced my childhood house so many times that he essentially rents it against interest - which is tax deductible. Since the mortgage inception in 1989, he's outperformed the market, but even that annualized rate is 8.6%. So, the marginal return over that period is something like a factor of 10. (1000%).
For comparison's sake, my wife's direct loans are at a relatively devastating 9.7% (can't refinance and get forgiveness).
As long as you aren't fearful of catastrophic economic straits and average in your investments over time (instead of plunging in at a time like now where the Q ratio indicates we're overbought), I'd willingly hold debt at 4% or less if it allowed me to allocate more capital to passively managed funds.