I took nearly 2000 words to get this across and you do capture the essence in four sentences. You are bang on with you conclusion, it is attraction and annoyance; he is vacillating between the two emotions.
In all honesty, I am really enjoying people's reaction to the kiss and his behavior. There is more to the man than his eagerness to rile her, he has more than one reason and she is the one giving them to him.