What's different between these 2 partial function objects?

Question

Precondition

I have these definitions:

def add(x,y): return (lambda x,y: x+y)(x,y)
def call(f,x,y): return f(x,y)

and these import aliases:

from multiprocessing import Pool as P;
from functools import partial as p;

Then I run this:

P(2).map(p(add,1),[2,3])

I can get a result: [3, 4]

But if I run this:

P(2).map(p(p(call,lambda x,y: x+y),1),[2,3])

it produces an error message.

• In Python version 2.7.5:

  Exception in thread Thread-8:
  Traceback (most recent call last):
    File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
      self.run()
    File "/usr/lib64/python2.7/threading.py", line 765, in run
      self.__target(*self.__args, **self.__kwargs)
    File "/usr/lib64/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
      put(task)
  PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
  

  (and then I have to press ^Z then kill %1 to close it)

• In Python version 3.6.8:

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
      return self._map_async(func, iterable, mapstar, chunksize).get()
    File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
      raise self._value
    File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
      put(task)
    File "/usr/lib64/python3.6/multiprocessing/connection.py", line 210, in send
      self._send_bytes(_ForkingPickler.dumps(obj))
    File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
      cls(buf, protocol).dump(obj)
  _pickle.PicklingError: Can't pickle <function <lambda> at 0x7f25b741cd90>: attribute lookup <lambda> on __main__ failed
  

  (better than python2: the REPL will give me the >>> again ...)

I've tried these, both are okay:

• run: p(add,1)(3) , get: 4
• run: p(p(call,lambda x,y: x+y),1)(3) , get: 4

Doubt

Now, I have my question:

I think they are same thing:

• (x,y) -> (lambda x,y: x+y)(x,y)
• partial((f,x,y) -> f(x,y) , lambda x,y: x+y)

Clearly they are in fact different. But how, and why?

Answer 1

A lambda function is an unnamed function. multiprocessing operates by pickle, and that serializes functions by storing their name, e.g. builtins.len:

>>> import pickle
>>> pickle.dumps(len)
b'\x80\x04\x95\x14\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x03len\x94\x93\x94.'

So when pickling a function built by lambda, the name lookup fails:

>>> pickle.dumps(lambda x: len(x))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fda31df4280>: attribute lookup <lambda> on __main__ failed

So the two functions have the same call behaviour, but not the same characteristics.