Functions as objects in Python: what exactly is stored in memory? -

i've been using python while solve practical problems, still don't have proper theoretical understanding of what's going on behind hood. example, i'm struggling understand how python manages treat functions objects. know functions objects of class 'function', 'call' method, , aware can make custom-made classes behave functions writing 'call method' them. can't figure out precisely gets stored in memory when new functions created, , how access information gets stored.

to experiment, wrote little script creates lots of function objects , stores them in list. noticed program used lot of memory.

funct_list = [] in range(10000000):     def funct(n):         return n +     funct_list.append(funct) 

my questions are:

• what precisely gets stored in ram when define new function object? storing details of how function implemented?

• if so, function object have attributes or methods allow me "inspect" (or possibly "alter retrospectively") way function behaves?

• maybe previous question circular, because methods of function object functions in own right...

• in code above, of ram used store "pointers" function objects in list. rest of ram presumably used store interesting stuff how function objects work. how ram distributed between these 2 purposes?

• suppose alter code snippet making function more complicated stuff. use more ram consequence? (i expect so. when altered definition of function filling body 1000 lines of junk, there didn't appear difference in amount of ram used up.)

i love find comprehensive reference this. whatever type google, can't seem find i'm looking for!

a function object's data divided 2 primary parts. parts same functions created same function definition stored in function's code object, while parts can change between functions created same function definition stored in function object.

the interesting part of function bytecode. core data structure says execute function. it's stored bytestring in function's code object, , can examine directly:

>>> def fib(i): ...     x, y = 0, 1 ...     _ in range(i): ...         x, y = y, x+y ...     return x ...  >>> fib.__code__.co_code b'd\x03\\\x02}\x01}\x02x\x1et\x00|\x00\x83\x01d\x00]\x12}\x03|\x02|\x01|\x02\x17\x00\x02\x00}\x01}\x02q\x1 2w\x00|\x01s\x00' 

...but it's not designed human-readable.

with enough knowledge of implementation details of python bytecode, parse yourself, describing take way long. instead, we'll use dis module disassemble bytecode us:

>>> import dis >>> dis.dis(fib)   2           0 load_const               3 ((0, 1))               2 unpack_sequence          2               4 store_fast               1 (x)               6 store_fast               2 (y)    3           8 setup_loop              30 (to 40)              10 load_global              0 (range)              12 load_fast                0 (i)              14 call_function            1              16 get_iter         >>   18 for_iter                18 (to 38)              20 store_fast               3 (_)   4          22 load_fast                2 (y)              24 load_fast                1 (x)              26 load_fast                2 (y)              28 binary_add              30 rot_two              32 store_fast               1 (x)              34 store_fast               2 (y)              36 jump_absolute           18         >>   38 pop_block   5     >>   40 load_fast                1 (x)              42 return_value 

there number of columns in output here, we're interested in 1 all_caps , columns right of that.

the all_caps column shows function's bytecode instructions. example, load_const loads constant value, , binary_add instruction add 2 objects +. next column, numbers, bytecode arguments. example, load_const 3 says load constant @ index 3 in code object's constants. these integers, , they're packed bytecode string along bytecode instructions. last column provides human-readable explanations of bytecode arguments, example, saying 3 in load_const 3 corresponds constant (0, 1), or 1 in store_fast 1 corresponds local variable x. information in column doesn't come bytecode string; it's resolved examining other parts of code object.

---

the rest of function object's data stuff needed resolve bytecode arguments, function's closure or global variable dict, , stuff exists because it's handy introspection, function's __name__.

if take @ python 3.6 function object struct definition @ c level:

typedef struct {     pyobject_head     pyobject *func_code;    /* code object, __code__ attribute */     pyobject *func_globals; /* dictionary (other mappings won't do) */     pyobject *func_defaults;    /* null or tuple */     pyobject *func_kwdefaults;  /* null or dict */     pyobject *func_closure; /* null or tuple of cell objects */     pyobject *func_doc;     /* __doc__ attribute, can */     pyobject *func_name;    /* __name__ attribute, string object */     pyobject *func_dict;    /* __dict__ attribute, dict or null */     pyobject *func_weakreflist; /* list of weak references */     pyobject *func_module;  /* __module__ attribute, can */     pyobject *func_annotations; /* annotations, dict or null */     pyobject *func_qualname;    /* qualified name */      /* invariant:      *     func_closure contains bindings func_code->co_freevars,      *     pytuple_size(func_closure) == pycode_getnumfree(func_code)      *     (func_closure may null if pycode_getnumfree(func_code) == 0).      */ } pyfunctionobject; 

we can see there's code object, , then

• the global variable dict,
• the default argument values,
• the keyword-only argument default values,
• the function's closure cells,
• the docstring,
• the name,
• the __dict__,
• the list of weak references function,
• the __module__,
• the annotations, and
• the __qualname__, qualified name

inside pyobject_head macro, there's type pointer , refcount/gc metadata.

we didn't have go straight c examine of - have looked @ dir , filtered out non-instance attributes, since of data available @ python level - struct definition provides nice, commented, uncluttered list.

you can examine code object struct definition too, contents aren't clear if you're not familiar code objects, i'm not going embed in post. i'll explain code objects.

the core component of code object bytestring of python bytecode instructions , arguments. examined 1 of earlier. in addition, code object contains things tuple of constants function refers to, , lot of other internal metadata required figure out how execute each instruction. not metadata - of comes function object - lot of it. of it, tuple of constants, understandable, , of it, co_flags (a bunch of internal flags) or co_stacksize (the size of stack used temporary values) more esoteric.