In 0.0, bubbles can be a powerful tool for catching at least the stupider bots. Even if they dock up when a non-friendly jumps into system they will often continue warping to the same location when you leave, get caught in the bubble, and when you come back they'll sit there thinking in their stupid little bot minds that they are safe in their POS or station. I expect that more sophisticated bots at this point are capable of recognizing and dealing with this tactic, however, as it has been well-publicized.
Local chatter, convos, or eve-mails can also be good tools for identifying bots, but people may not respond if they speak a different language or are just shy. It's a good rule-out for people who are NOT bots, however.
Being logged on 23/7 is sometimes a good cue, but specifically being logged on and active. A lot of people will just leave themselves perma-logged-on to make their prime hours difficult to trace, but no human can actually do things in EVE for 23 hours a day, 7 days a week.
Beyond that it gets difficult. If you've got a very good understanding of how bots work and what data they read from EVE, you could try to find some way to mess with their input and confound them, but you're as likely to do something that would be considered an "Exploit" doing that as the bots themselves.
The only ones with the tools to truly, unambiguously identify bots are CCP, but they haven't been great about enforcing anything. If, based on the above, you suspect that someone is a bot you might as well report it since it can do you no harm and the GMs might take action, but realistically it's a crapshoot.
Oh, and if you have someone you're pretty sure is a bot, report them to their alliance, even if the alliance is hostile to you. Often an entire corp will be bots so it's not worth reporting at the corp level, but alliances have to be a little on guard because if there is any traceable evidence that they are allowing known bots CCP can screw them proper. Also, at least in 0.0, blues will be much more able to identify bots because most bots won't try to hide when blues show up and they can be more readily observed.
+1 and +fav, because this question is asked very often and i’ve discussed this issue with corp members as well.
A very interesting question