Soft Breakup with Financial Advisor

I have my investment account, old 401k, and Roths consolidated under a financial advisor who gives me honest good advice so far on taxes and as a first generation with investments I appreciate his guidance. However, his fees do hurt my returns.

How wrong would it be to keep the investments in have with him but begin funneling any additional investments to a vanguard account -- keep the advice, keep paying some, but utilmately end up with most of my money outside of his practice?