Oh, for sure you can get sued for that, because one factor in whether or not that can happen is whether people are recognizable. And people will be looking for themselves in a memoir.

One of the lawyers I mention of the story gives a tip late in the story that applies to memoirs, too: You have to make really sure the person won't be recognized, either by that person OR anyone else.