From Defending the Honor of Hashem (#E-238) by Rabbi Avigdor Miller
Q: Is it wrong to move out of town in order to join a kollel where we can be be involved in educating people?
A: If you do it for a short time, maybe. But you must know that you are sacrificing your family. I know from experience, if your family will remain out of town in a negative environment, who knows what can happen chas vesholom.
If you move into a frum neighborhood, your family is transformed. The sons and daughters go to the very best schools, like the Yeshiva of Brooklyn. The people who have TV are not allowed to send their children there, they won’t tell you about TV in those places.
In the long run, it’s going to make a difference in what your family will look like
Who needs to travel to Manchester, or to Gateshead or to Eretz Yisroel? You have it right here. In the frum neighborhoods you have to best institutions, beautiful yeshivas. If you are worried about your family, you have to make sure that your family grows up in a good neighborhood.
As far as how to be mukarev Jews who are way out there, that’s a question. But it’s no question that your life comes first. It’s so important; you cannot know what risks you’re taking when you go away from a frum neighbourhood and settle among half Jews. Even if the Jews are shomrei mitzvos (Torah observant), but they’re weak in emunah (faith) — they have gentile names, they all have TV’s — it’s a terrible environment. They’re all sinking into the slump of assimilation.
You better think it over, because in the long run, it’s going to make a difference in what your family will look like.