I tried to practice with SGI while no longer accepting many of their teachings. I resisted quitting because I felt I benefited from the community. At some point, I couldn't take it anymore and quit. The day I did so, its hard to qualify, but my practice was freed. I still look on people who practice with SGI fondly, but I don't share their views, so its hard for me to abide them. I'm just not an Ikeda fanatic, I don't buy their Buddhist humanism spin, I don't interpret Nichiren the way they do, and its hard to talk about Buddhism in the terms they use. I still am concerned how I will introduce my children to Buddhist practice without a community - its always just going to be some weird thing Dad does. I don't know the answer to that, but I also can't accept the options around me.
Check out ION Buddhism's website. They have an automated app on their site where you can sign up and be put in touch with other independents in your area. Who knows, there may be another person feeling the way you do living next door. You could get together and just chant and study?