python - Fastest way to find non-finite values -


this inspired by: python: combined masking in numpy.

the task create boolean array of values not finite. example:

>>> arr = np.array([0, 2, np.inf, -np.inf, np.nan]) >>> ~np.isfinite(arr) array([false, false,  true,  true,  true], dtype=bool)

to me, seems fastest way find non-finite values, seems there faster way. np.isnan(arr - arr) should same:

>>> np.isnan(arr - arr) array([false, false,  true,  true,  true], dtype=bool)

timing see twice fast!

arr = np.random.rand(100000)  %timeit ~np.isfinite(arr) 10000 loops, best of 3: 198 µs per loop  %timeit np.isnan(arr - arr) 10000 loops, best of 3: 85.8 µs per loop

so question twofold:

  1. why np.isnan(arr - arr) trick faster "obvious" ~np.isfinite(arr) version? there input not work for?

  2. is there faster way find non-finite values?

that's hard answer because np.isnan , np.isfinite can use different c functions depending on build. , depending on performance (which may depend on compiler, system , how numpy built) of these c functions timings different.

the ufuncs both refer built-in npy_ func (source (1.11.3)):

/**begin repeat1  * #kind = isnan, isinf, isfinite, signbit, copysign, nextafter, spacing#  * #func = npy_isnan, npy_isinf, npy_isfinite, npy_signbit, npy_copysign, nextafter, spacing#  **/

and these functions defined based on presence of compile time constants (source (1.11.3)):

/* use builtins avoid function calls in tight loops  * available if npy_config.h available (= numpys own build) */ #if have___builtin_isnan     #define npy_isnan(x) __builtin_isnan(x) #else     #ifndef npy_have_decl_isnan         #define npy_isnan(x) ((x) != (x))     #else         #if defined(_msc_ver) && (_msc_ver < 1900)             #define npy_isnan(x) _isnan((x))         #else             #define npy_isnan(x) isnan(x)         #endif     #endif #endif  /* available if npy_config.h available (= numpys own build) */ #if have___builtin_isfinite     #define npy_isfinite(x) __builtin_isfinite(x) #else     #ifndef npy_have_decl_isfinite         #ifdef _msc_ver             #define npy_isfinite(x) _finite((x))         #else             #define npy_isfinite(x) !npy_isnan((x) + (-x))         #endif     #else         #define npy_isfinite(x) isfinite((x))     #endif #endif

so might in case np.isfinite has (much) more work np.isnan. it's equally on computer or build np.isfinite faster or both equally fast.

so, there not hard rule "fastest way" is. depends on many factors. go np.isfinite because can faster (and isn't slower in case) , makes intention clearer.

just in case you're optimizing performance, can negating in-place. might decrease time , memory avoiding 1 temporary array:

import numpy np arr = np.random.rand(1000000)  def isnotfinite(arr):     res = np.isfinite(arr)     np.bitwise_not(res, out=res)  # in-place     return res  np.testing.assert_array_equal(~np.isfinite(arr), isnotfinite(arr)) np.testing.assert_array_equal(~np.isfinite(arr), np.isnan(arr - arr))  %timeit ~np.isfinite(arr) # 3.73 ms ± 4.16 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) %timeit isnotfinite(arr) # 2.41 ms ± 29.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) %timeit np.isnan(arr - arr) # 12.5 ms ± 772 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

note np.isnan solution much slower on computer (windows 10 64bit python 3.5 numpy 1.13.1 anaconda build)


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more