Ticket #1502 (closed bug: fixed)

Opened 7 years ago

Last modified 6 years ago

channels and tasklets not serializable under memory session manager, which breaks component.Task - at least

Reported by: pjnagel Owned by: apoirier
Priority: major Component: session managers
Version: 0.3.0 Keywords:
Cc:

Description

The following little app works fine under the default pickling session manager, but fails when using the memory session manager:

from nagare import component, util

class Oops(component.Task):
    def go(self, comp):
        comp.call(util.Confirm('Please click'))
        comp.call(util.Confirm('This is never reached under memory session type'))

def app():
    return Oops()

After the first click, the following stacktrace is dumped:

Exception happened during processing of request from ('127.0.0.1', 58311)
Traceback (most recent call last):
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/paste/httpserver.py", line 1056, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/usr/local/stackless/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/stackless/lib/python2.7/SocketServer.py", line 639, in __init__
    self.handle()
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/paste/httpserver.py", line 432, in handle
    BaseHTTPRequestHandler.handle(self)
  File "/usr/local/stackless/lib/python2.7/BaseHTTPServer.py", line 337, in handle
    self.handle_one_request()
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/paste/httpserver.py", line 427, in handle_one_request
    self.wsgi_execute()
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/paste/httpserver.py", line 287, in wsgi_execute
    self.wsgi_start_response)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/paste/urlmap.py", line 202, in __call__
    return app(environ, start_response)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/wsgi.py", line 379, in __call__
    response = self.on_exception(request, response)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/wsgi.py", line 345, in __call__
    session = self.sessions.get(request, response, xhr_request)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/sessions/common.py", line 209, in get
    session_secure_id = state.get()
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/sessions/common.py", line 83, in get
    (new_state_id, self.lock, self.secure_id, self.data) = self.sessions_manager.get_state(self.session_id, self.state_id, self.use_same_state)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/sessions/common.py", line 279, in get_state
    data = self.deserialize(session_data, state_data)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/site-packages/nagare/sessions/memory_sessions.py", line 208, in deserialize
    return copy.deepcopy(state_data) if self.states_history else state_data
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/local/stackless/lib/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/home/pnagel/Projects/eots/.tox/stackless-py27/lib/python2.7/copy_reg.py", line 100, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(NotImplementedType) is not safe, use NotImplementedType.__new__()

By instrumenting /usr/local/stackless/lib/python2.7/copy.py to add "print x" to the start of _reconstruct(), so that the object being deepcopied is printed as the deepcopy traverses the object graph, we get the following output:

<component at 3063510 on object <nagare.util.Confirm object at 0x30689d0>>
<stackless.channel object at 0x3057230>
<stackless.tasklet object at 0x3067590>
<frame object at 0x3138150>
<module 'stackless' (built-in)>
Type help() for interactive help, or help(object) for help about object.
Copyright (c) 2001-2010 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved.
Use exit() or Ctrl-D (i.e. EOF) to exit
    Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information.
See http://www.python.org/2.7/license.html
Use quit() or Ctrl-D (i.e. EOF) to exit
NotImplemented

As I understand it, this is a seriously limits the viability of the memory session manager for Nagare.

The proximate problem is that channels cannot be deepcopied, which is serious enough since they play a central role in Stackless.

But the deeper problem is that pickling and deepcopying depend on totally different code paths. Problematic Third party library objects may override getstate and setstate to make themselves pickleable, but then they still need to override deepcopy to render them deepcopyable.

And since the Stackless project has a deep commitment to pickling, it is likely that they will make pickling work for everything but that the effort to make deepcopying work will fall on the shoulders of the Nagare project itself. In other words, having created the memory session manager, the Nagare project is now at risk of playing whack-a-mole with deepcopying bugs in third party code.

It might be better to just remove the memory session manager totally to conserve development energy. If there is a deficiency with the pickling session manager, which was the reason that the memory session manager was created in the first place, then it might be more beneficial to rather expend energy on solving and mitigating those problems in the context of the pickling session manager.

Change History

comment:1 in reply to: ↑ description Changed 7 years ago by apoirier

Hi,

Thanks for this detailed report,

The following little app works fine under the default pickling session manager, but fails when using the memory session manager:

If you want to keep the sessions in memory, use the standalone session manager. This manager stores a pickle of the sessions, so without the problems you describe (http://www.nagare.org/trac/wiki/PublisherConfiguration#sessions-section).

The memory session manager is a special manager only created for the purpose of the Nagare IDE application, where not even an history of the sessions is stored.

...
But the deeper problem is that pickling and deepcopying depend on totally different code paths. Problematic Third party library objects may override getstate and setstate to make themselves pickleable, but then they still need to override deepcopy to render them deepcopyable.

And since the Stackless project has a deep commitment to pickling, it is likely that they will make pickling work for everything but that the effort to make deepcopying work will fall on the shoulders of the Nagare project itself. In other words, having created the memory session manager, the Nagare project is now at risk of playing whack-a-mole with deepcopying bugs in third party code.

It might be better to just remove the memory session manager totally to conserve development energy. If there is a deficiency with the pickling session manager, which was the reason that the memory session manager was created in the first place, then it might be more beneficial to rather expend energy on solving and mitigating those problems in the context of the pickling session manager.

Right. To avoid any confusions, now I'm thinking the best solution would be to move this memory manager to the nagare.ide package.

comment:2 Changed 6 years ago by Alain Poirier <alain.poirier@…>

In [a2a7872c6b83ad85afdaa3086db73d9f30c21261/core]:

Pure memory sessions moved to nagare.ide project (see #1502)

comment:3 Changed 6 years ago by Alain Poirier <alain.poirier@…>

  • Status changed from new to closed
  • Resolution set to fixed

In [6c4602536479b285ae028deee6d1b5806c93d797/ide]:

Fixes #1502: pure memory sessions moved to nagare.ide project

Note: See TracTickets for help on using tickets.