What's Fair Game When Writing?

In the specific sense of, "What aspects of your life and the lives of the people around you are fair game for including in your writing, be it fiction or nonfiction?"

Based on a great deal of what I've read, the answer for many established writers is: EVERYONE AND EVERYTHING!

It does seem to me that a lot of the most powerful writing is greatly informed by the personal life and relationships of the author. On the other hand, how often do the writers have the explicit or implicit permission of those they write about to use their personal interactions as the basis for a story?

Now, a lot of people will just say, "if you're talking about your own experiences with someone else, events and people from your own life, then you own those experiences and you should feel free to write about them." And I agree with that in principle. Everything is supposed to be grist for the mill.

But what about the effect of such borrowing from real life on the people who were borrowed from? Any good character has to have a range of emotions, motivations, and thoughts--in other words, they need to have some flaws and make some mistakes. Inherent in any such depiction is the chance for the person being portrayed to be offended, perhaps due to a misinterpretation or maybe because an uncomfortable truth is being exposed. At the very least it raises questions about how the author views that person.

If you care more about the honesty and depth of the work you're writing than about the feelings of the people you include, then this isn't an issue. But I can't say that for myself. There are parts of my life and people in my life that I won't write about or even try to consciously draw inspiration from. And I feel that's a healthy approach, both in terms of my own mental well-being and the continued enjoyment of relationships that matter to me.

But I don't know if this implies that I don't care enough about the process of writing. I do think it suggests one reason why I'm so engrossed in speculative fiction--there's more room for imagination and variations in setting to conceal the source of characterizations and events. That's a more comfortable zone for me to operate in.