My mum is in a similar situation at the moment where a neighbours cat has 'adopted' her and will not take no for an answer. If they don't let him in he damages their patio door with incessant scratching and is just incredibly miserable unless they let him in
He lives in a household where there are 2 other cats and the owner says that he doesn't enjoy their company at all. So we think the reason he likes going to my mums is that he is the only cat there, he gets all the attention and has a peaceful day there, until they throw him out at bed time (which they are now starting to feel guilty about as its so cold).
Is it possible Bobby has taken a disliking to your other 2 cats? Does this other lady have any pets? I really feel for you as you are obviously a very loving cat-mum and are giving him all you can in terms of love and fulfilling his needs. Do keep us posted.