text mining - Lemmatization using txt file with lemmes in R -


i use external txt file polish lemmas structured follows: (source lemmas many other languages http://www.lexiconista.com/datasets/lemmatization/)

abadan  abadanem abadan  abadanie abadan  abadanowi abadan  abadanu abadańczyk  abadańczycy abadańczyk  abadańczyka abadańczyk  abadańczykach abadańczyk  abadańczykami abadańczyk  abadańczyki abadańczyk  abadańczykiem abadańczyk  abadańczykom abadańczyk  abadańczyków abadańczyk  abadańczykowi abadańczyk  abadańczyku abadanka    abadance abadanka    abadanek abadanka    abadanką abadanka    abadankach abadanka    abadankami

what packages , syntax, allow me use such txt database lemmatize bag of words. realize, english there wordnet, there no luck use functionality rare languages.

if not, can database converted useful package provides lemmatization? perhaps converting wide form? instance, form used free antconc concordancer, (http://www.laurenceanthony.net/software/antconc/)

abadan -> abadanem, abadanie, abadanowi, abadanu abadańczyk -> abadańczycy, abadańczyka, abadańczykach  etc.

in brief: how can lemmatization lemmas in txt file done in of known cran r text mining packages ? if so, how format such txt file?

update: dear @dmitriyselivanov got rid of diacritical marks, apply on tm corpus "docs"

docs <- tm_map(docs, function(x) lemma_tokenizer(x, lemma_hashmap="lemma_hm"))

and tried tokenizer

lemmatokenizer <- function(x) lemma_tokenizer(x, lemma_hashmap="lemma_hm")  docstdm <-   documenttermmatrix(docs, control = list(wordlengths = c(4, 25), tokenize=lemmatokenizer))

it throws @ me error: 

 error in lemma_hashmap[[tokens]] :    attempt select more 1 element in vectorindex

the function works vector of texts charm though.

my guess here nothing text-mining packages task. need replace word in second column word in first column. can creating hashmap (for example https://github.com/nathan-russell/hashmap).

below example of how can create "lemmatizing" tokenizer can use in text2vec (and guess quanteda well).

contributions in order create such "lemmatizing" package welcome - useful.

library(hashmap) library(data.table) txt =    "abadan  abadanem   abadan  abadanie   abadan  abadanowi   abadan  abadanu   abadańczyk  abadańczycy   abadańczyk  abadańczykach   abadańczyk  abadańczykami   " dt = fread(txt, header = f, col.names = c("lemma", "word")) lemma_hm = hashmap(dt$word, dt$lemma)  lemma_hm[["abadanu"]] #"abadan"   lemma_tokenizer = function(x, lemma_hashmap,                             tokenizer = text2vec::word_tokenizer) {   tokens_list = tokenizer(x)   for(i in seq_along(tokens_list)) {     tokens = tokens_list[[i]]     replacements = lemma_hashmap[[tokens]]     ind = !is.na(replacements)     tokens_list[[i]][ind] = replacements[ind]   }   tokens_list } texts = c("abadanowi abadańczykach outofvocabulary",            "abadańczyk abadan outofvocabulary") lemma_tokenizer(texts, lemma_hm)  #[[1]] #[1] "abadan"          "abadańczyk"      "outofvocabulary" #[[2]] #[1] "abadańczyk"      "abadan"          "outofvocabulary"

Comments

Post a Comment

Popular posts from this blog

What is happening when Matlab is starting a "parallel pool"? -

running parallel cpu processes in matlab starts command parpool() according documentation , function: [creates] special job on pool of workers, , [connects] matlab client parallel pool. this function takes bit of time execute, on order of 30 seconds. in other multi-cpu paradigms openmp, parallel execution seems totally transparent -- i've never noticed behavior analogous matlab (granted i'm not experienced parallel programming). so, happening between time parpool() called , when finishes executing? takes long? parallel computing toolbox enables run matlab code in parallel using several different paradigms (e.g. jobs , tasks, parfor , spmd , parfeval , batch processing), , run either locally (parallelised across cores in local machine) or remotely (parallelised across machines in cluster - either 1 own, or 1 in cloud). in of these cases, code run on matlab workers , copies of matlab without interactive desktop. if you're intending run on remo...
Read more

angular - DownloadURL return null in below code -

when change file updated in firebase storage when try downloadurl show null. filechange(event: any) { const imagefolder: string = this.employee.id; const filelist: filelist = event.target.files; const file: file = filelist[0]; const storageref = firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file); console.log(storageref.snapshot.downloadurl); } the put() asynchronous. if want url after file uploaded have this: firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file).then((snapshot) => { storageref = snapshot.downloadurl: console.log(snapshot.downloadurl); });
Read more

php - Cannot override Laravel Spark authentication with own implementation -

during login process of laravel spark need request token of remote api. api stores exact same users , passwords laravel spark application. therefore need username , non hashed password of user during authentication process. i thought overriding authenticated method solution problem. in routes/web.php override post /login endpoint , pointing endpoint own logincontroller: <?php namespace app\http\controllers\auth; use illuminate\support\facades\log; use illuminate\http\request; use laravel\spark\http\controllers\auth\logincontroller sparklogincontroller; class logincontroller extends sparklogincontroller { /** * create new login controller instance. * * @return void */ public function __construct() { parent::__construct(); } /** * handle successful authentication attempt. * * @param request $request * @param \illuminate\contracts\auth\authenticatable $user * @return response */ public fu...
Read more