what's the best open protocol for chat room software?
I'm comparing between IRC, XMPP and a third proprietary server I have.
what are the pros and cons of each and suggest others if you know any mentionable ones.
Just go for XMPP.
There are servers known to scale big, it can work with a web client with minimum hassle (if that's what you need) with great libraries to support it, and, the XEP-45 specifies exactly how the clients and servers are supposed to operate.
I know this is a very old post, and the op will likely not see the reply (or maybe even care any longer), but I would deploy IRC over XMPP personally. The big reason for this is the amount of memory that XMPP servers occupy as more and more XMPP clients connect. On a single instance, I have not seen rock-solid reliability when more than about 200 users connect, and actively start using the service (XMPP MUC, file transfers, client-to-client encryption, etc).
Contrasted with IRC, I've personally seen thousands of connections to a single instance, and the memory utilization is usually 60-70% less. The IRC server does not need to parse the messages passed from the client, file transfers are client-to-client via DCC, and the design is much, much simpler than the XML-based XMPP.
You may run into netsplits, if you run multiple servers, and allow clients to choose which server to connect to. But how is an unstable IRC server any different than an unstable XMPP server? And if an IRC server netsplits, it doesn't take every client offline- only those connected to it. So the impact is minimal, compared to XMPP (unless you're doing s2s).
Regardless, the IRC protocol is more simplistic in nature, can handle orders of magnitude more client connections than XMPP for the same memory utilization, uses less bandwidth on the wire, doesn't require authentication (although you can add this feature), etc.
It seems I'm not the only one who feels this way: http://web.archive.org/web/20131128110041/http://laltromondo.dynalias.net:80/~iki/informatica/soft/xmpp_vs_irc/.
For future readers of this post, consider IRC. It may just save you some headache later down the road.
You might want to consider PSYC.
It's like the blend of the best features of XMPP and IRC, and then some more :
Since it can provide most of what you would expect from IRC, you could consider psyced before starting an IRC network. PSYC solves IRC's political, authentication and scalability issues and comes with a huge choice of extra features that go beyond what you would expect from an IRC server, still psyced can be used with all the IRC clients you are accustomed with. Up to a certain degree, PSYC and psyced also support things you would expect from Jabber, and even XMPP itself, but you may run into problems depending on your special requirements. psyced is also an XMPP server and gateway at the same time. The PSYC protocol is capable of delivering encrypted data natively, although we're only going to start using that in 2012.