Text File

Persistent Modules

Document Overview

This document seeks to capture technical information about persistent modules to guide and document their design.


These goals largely come from Zope 3. It would be worth while considering other applications.

Use cases

Edge cases


Fundamental dilema

Python modules were not designed to change at run time. The source of a Python module normally doesn't change while a Python program is running. There is a crude reload tool that allows modules to be manually reloaded to handle source changes.

Python modules contain mutable state. A module has a dictionary that may be mutated by application code. It may contain mutable data that is modified at run time. This is typeically used to implement global registries.

When a module is reloaded, it is reexecuted with a dictionary that includes the results of the previous execution.

Programs using the ZODB may be said to have logical lifetimes that exceed the lifetimes of individual processes. In addition, the program might exist as multiple individual processes with overlapping run-times.

The lifetime of a persistent program is long enough that it is likely that module source code will change during the life time of the program.


Should the state of a module be represented soley by the module source?

Consider the possibilities:

  1. Module state is represented soley by it's source.
  1. Module state isn't represented soley by it's source.
  • It would become possible to allow mutable data, such as registries in persistent modules.

  • It could be very difficult to see what a module's state is. If a module contained mutable data, you'd need some way to get to that data so you could inspect and manipulate it.

  • When a module is synchronized to the file system, you'd need to syncronize it's source and you'd also need to synchronize it's contents in some way. Synchronization of the contents could be done using an XML pickle, but management of the data using file-system-based tools would be cumbersome.

    You'd end up with data duplicated between the two representations. It would be cumbersome to manage the duplicated data in a consistent way.

  1. Module state is represented soley by it's source, but allow additional meta data.

    This is the same as option A, except we support meta-data management. The meta data could include dependency information. We'd keep track of external usage (import) of module variables to influence whether deletion of the module or defined variables is allowed, or whether to issue warnings when variables are deleted.

    Note that the management of the meta data need not be the responsibility of the module. This could be done via some application-defined facility, in which case, the module facility would need to provide an api for implimenting hooks for managing this information.

Special cases

This section contains examples that may introduce challenges for persistent modules or that might motivate or highlight issues described above,