On Authority is a pointed response to the thought-terminating pejorative of “authoritarian.” If you haven’t read it, you should! It’s short and important =)
You want a change in society? You don’t care that fascists don’t want that change? Then you’re an authoritarian. Everything after that is a matter of degree.
If you’re willing to do something about that wish, then you’re an authoritarian too.
And FWIW, casting it in personal terms makes that wish tautological. No one wishes for a system where they have to cater to the whims of someone else. I’m sure you meant to say “no one has to cater to anyone else’s whims,” and that’s all well and good. Still authoritarian if you’re willing to see anything at all done to overcome the desires of the powerful to maintain the status quo (and in case it’s not obvious, that’s a good thing to be authoritarian about).