python - How to make casefold() work on certain Arabic unicodes -


i've got issues detecting "equality" in python 2.7 of arabic pairs of words:

  1. أكثر vs اكثر
  2. قائمة vs قائمه
  3. إنشاء vs انشاء

the elements of each pair not identical, written different cases. useful analogy me (i don't know arabic) word vs word. not identical, if lowercase both of them, i'll obtain word vs word, identical. that's want obtain these 3 pairs of arabic words.

i'm going exemplify tried using first pair (1. أكثر vs اكثر). way, meaning of both arabic words first pair "menu" "more", have different cases (as parallel: menu vs menu more vs more). don't know arabic @ nor arabic rules, if knows arabic can confirm words "identical" great.

str1 = u'أكثر' str2 = u'اكثر'

so i'm trying bring str1 , str2 same form (if possible), want function produce same output both strings:

transform(str1) == transform(str2)

in english can achieved easily:

a = u'more' b = u'more'  def transform(text):     return text.lower()  >>> transform(a) == transform(b) >>> true

but, of course, doesn't work arabic there no such things lower case or upper case.

>>> str1 u'\u0623\u0643\u062b\u0631'  >>> str2 u'\u0627\u0643\u062b\u0631'

note first character differs in unicode representation.

i normalized strings using:

import unicodedata  >>> n_str1 = unicodedata.normalize('nfkd', str1) >>> n_str2 = unicodedata.normalize('nfkd', str2)  >>> n_str1 u'\u0627\u0654\u0643\u062b\u0631'  >>> n_str2 u'\u0627\u0643\u062b\u0631'

as noticed:

>>> n_str1 == n_str2 false

after that, tried use unicode.casefold() isn't available in python 2. i've installed py2casefold library didn't manage obtain equality between strings. tried use python 3's unicode.casefold() without luck:

>>> str1.casefold() == str2.casefold() false  >>> n_str1.casefold() == n_str2.casefold() false

a solution in python 2 perfect, great in python 3 too.

thank you.

these words not identical: u'أكثر' , u'اكثر' not same. first letter in first word has letter alif hamazah on top of it, perhaps couldn't notice due small size of glyph:

alif hamaza

the first letter in second word, however, alif *(from right-to-left):

alif

and hence don't compare equal. each of these letters represented own unicode character code point. don't compare equal perspective of language too: 

>>> u'أكثر'; u'اكثر' u'\u0623\u0643\u062b\u0631' u'\u0627\u0643\u062b\u0631'

they not identical, if lowercase both of them, i'll obtain word vs word, identical. that's want obtain these 3 pairs of arabic words.

there's no lower or upper case in arabic. words have in hands not same, have different letters. of words have correct spelling while others have incorrect spelling. may seem same, arabic readers may consider them same, language freaks, they're not same. convey meaning, list of arabic words in english looks this:

1- more, moore

2- menu, manu

3- establish, estblish

i'm going exemplify tried using first pair (1. أكثر vs اكثر). way, meaning of both arabic words first pair "menu", have different cases (as parallel: menu vs menu)

no, أكثر means more. second pair means menu, there's no such thing menu or menu in arabic. couldn't delve details, because off topic.


Comments

Post a Comment

Popular posts from this blog

What is happening when Matlab is starting a "parallel pool"? -

running parallel cpu processes in matlab starts command parpool() according documentation , function: [creates] special job on pool of workers, , [connects] matlab client parallel pool. this function takes bit of time execute, on order of 30 seconds. in other multi-cpu paradigms openmp, parallel execution seems totally transparent -- i've never noticed behavior analogous matlab (granted i'm not experienced parallel programming). so, happening between time parpool() called , when finishes executing? takes long? parallel computing toolbox enables run matlab code in parallel using several different paradigms (e.g. jobs , tasks, parfor , spmd , parfeval , batch processing), , run either locally (parallelised across cores in local machine) or remotely (parallelised across machines in cluster - either 1 own, or 1 in cloud). in of these cases, code run on matlab workers , copies of matlab without interactive desktop. if you're intending run on remo...
Read more

angular - DownloadURL return null in below code -

when change file updated in firebase storage when try downloadurl show null. filechange(event: any) { const imagefolder: string = this.employee.id; const filelist: filelist = event.target.files; const file: file = filelist[0]; const storageref = firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file); console.log(storageref.snapshot.downloadurl); } the put() asynchronous. if want url after file uploaded have this: firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file).then((snapshot) => { storageref = snapshot.downloadurl: console.log(snapshot.downloadurl); });
Read more

php - Cannot override Laravel Spark authentication with own implementation -

during login process of laravel spark need request token of remote api. api stores exact same users , passwords laravel spark application. therefore need username , non hashed password of user during authentication process. i thought overriding authenticated method solution problem. in routes/web.php override post /login endpoint , pointing endpoint own logincontroller: <?php namespace app\http\controllers\auth; use illuminate\support\facades\log; use illuminate\http\request; use laravel\spark\http\controllers\auth\logincontroller sparklogincontroller; class logincontroller extends sparklogincontroller { /** * create new login controller instance. * * @return void */ public function __construct() { parent::__construct(); } /** * handle successful authentication attempt. * * @param request $request * @param \illuminate\contracts\auth\authenticatable $user * @return response */ public fu...
Read more