how to create a global variable in spark that can be used across all nodes -


this question has answer here:

i need use variable across nodes in spark yarn cluster. broadcast variables in spark immutable so, not useful in case. need similar approach supports both read , write.

regards, sorabh

you cannot, spark built on principle of immutability, in fact distributed framework works leveraging concepts of immutability.

here similar question , beautiful explanation : how define global read\write variables in spark


Comments

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

reflection - How to access the object-members of an object declaration in kotlin -

php - Doctrine Query Builder Error on Join: [Syntax Error] line 0, col 87: Error: Expected Literal, got 'JOIN' -