Self-inquiry is asking yourself, repeatedly, "Who am I?"
In my understanding, this has two aims:
The first aim is stabilizing the mind on the sense of being-ness. The question, "Who am I?" repeatedly brings the mind to that point, until it sticks there. It is like taking (instead of the breath) the feeling of being as a concentration object.
With carefully stabilized attention, I search with my “inner peripheral vision.” I tenuously, manage to find and place my attention on my sense of “I.” I hold my peripheral attention on that “I sensation” for long seconds. I exhaust it, deplete it. Finally, I relax. And that sense of “I” is gone. I feel hollow, like a shell. Self-referential cognition goes into a void, or “grounds out.” Any thought that has “me” involved hits that void and doesn't continue on. It's very scary. “Oh, shit,” what's left thinks. “Now I've done it,” it says on autopilot. The part of me on autopilot figures that it'll probably fade. And it does, after ten minutes. I decide not to try that ever again.
The second aim, which may simply be a side-effect of the first, is achieveing a state that is effortlessly free of verbal thought. This kind of awakening. This occurs via asking "Who am I?" which tends to clear the mind of thought. It may also be the case that once the mind is stabilized on the sense of being, self-referential thought no longer has that emotional pull, nothing to grab attention with, and thus ceases.
—lately, I have been relating to the practice in another way, as an easily repeated insight experience. Asking "Who am I?" empties the mind of thought, for at least a moment. In that moment between thoughts, you notice that it's a pretty nice place to hang out. Less drama, less suffering. This inclines the mind in that direction. Iterate and you end up in the thought free state.
Gary Weber, who used self-inquiry to awaken, relates to it similarly:
The brain is on "your" side. The brain "likes" lower levels of stress, confusion, anxiety, fear, etc. If it sees this state repeatedly, it will "all by itself" work out the neural refunctionalizing to produce it. This is a consistent report.
—taken from this page
The thought 'who am I?' will destroy all other thoughts, and like the stick used for stirring the burning pyre, it will itself in the end get destroyed. Then, there will arise Self-realization.
When other thoughts arise, one should not pursue them, but should inquire: 'To whom do they arise?' It does not matter how many thoughts arise. As each thought arises, one should inquire with diligence, “To whom has this thought arisen?”. The answer that would emerge would be “To me”. Thereupon if one inquires “Who am I?”, the mind will go back to its source; and the thought that arose will become quiescent. With repeated practice in this manner, the mind will develop the skill to stay in its source.
All the texts say that in order to gain release one should render the mind quiescent; therefore their conclusive teaching is that the mind should be rendered quiescent; once this has been understood there is no need for endless reading. In order to quieten the mind one has only to inquire within oneself what one's Self is; how could this search be done in books? One should know one's Self with one's own eye of wisdom.
—Ramana Maharshi, Who Am I?