I'm not retired, but I moved to one of those "best places retire" because I fell in love with it while serving as the writer-in-residence at an arts center.
Although I live in a small town, not a city, almost everything you say resonated, particularly the connecting-flights issue. The nearest city had no direct flights to New York when I moved here, which meant that every trip there took almost an entire day. You couldn't take a quick weekend trip to a city because you had to spend a day in transit each way and needed at least four days to get your two-day weekend.
My life here has advantages that make it worthwhile, but I can see why it might be less so for others.