An important issue was raised in comments below, though exhaustion prevented me from pursuing the point at the time. The issue: should those Home Educators who stand up for the right not to have their homes invaded by Local Authorities expect support from those HEors who do have home visits? Can these two sets of people work together against an increasingly intrusive government?
Perhaps I don't know the right crowd. Perhaps they cross the street to avoid me, but all the HEors I know who do actually have visits, accept them only grudgingly. They would far rather not have them, and they therefore support all activism to prevent LAs from being able to insist on them in situations where there is no reason to think that a suitable education is not being provided. This makes it very possible to work together, I think.
Tell me what I'm missing!