Python unicode equal comparison failed

This question is linked to Searching for Unicode characters in Python

I read unicode text file using python codecs

codecs.open('story.txt', 'rb', 'utf-8-sig')

And was trying to search strings in it. But i'm getting the following warning.

UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

Is there any special way of unicode string comparison ?

Answers


You may use the == operator to compare unicode objects for equality.

>>> s1 = u'Hello'
>>> s2 = unicode("Hello")
>>> type(s1), type(s2)
(<type 'unicode'>, <type 'unicode'>)
>>> s1==s2
True
>>> 
>>> s3='Hello'.decode('utf-8')
>>> type(s3)
<type 'unicode'>
>>> s1==s3
True
>>> 

But, your error message indicates that you aren't comparing unicode objects. You are probably comparing a unicode object to a str object, like so:

>>> u'Hello' == 'Hello'
True
>>> u'Hello' == '\x81\x01'
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
False

See how I have attempted to compare a unicode object against a string which does not represent a valid UTF8 encoding.

Your program, I suppose, is comparing unicode objects with str objects, and the contents of a str object is not a valid UTF8 encoding. This seems likely the result of you (the programmer) not knowing which variable holds unicide, which variable holds UTF8 and which variable holds the bytes read in from a file.

I recommend http://nedbatchelder.com/text/unipain.html, especially the advice to create a "Unicode Sandwich."


Need Your Help

Efficient Clojure workflow?

clojure

I am developing a pet project with Clojure, but wonder if I can speed up my workflow a bit.

MongoDB mongorestore and existing collection with records

mongodb mongorestore

I need to import (restore) a collection generated with mongodump into an existing database and I'd like the records to be merged into the existing collection.