Title

Front End Logging

Author

Nathan Addy, Koichi Takahashi

Date

August 29, 2007

Status

Proposed

Abstract

We propose a new Logger architecture for E-Cell 4, which is based in the front-end (Python based). In this proposal, Logger objects will be created by the user in the python front-end, which will then create and manage back-end callback objects that are connected to the simulation objects whose values they record. At times when state should be recorded, these callbacks will be executed, which will have the effect of inducing the front-end loggers to record the values of their associated state variables.

Description

Introduction

In E-Cell Version 4, a front-end logging infrastructure is proposed. Loggers will be created by users in the front-end; when created, they will create a callback class in the back-end and register it with the simulation object they are to log. Each Stepper will maintain a set of the callbacks associated with the variables in its mutator set. At the end of each of its cycles of stepping, it will execute each of those callbacks through their initiateLogging() member functions, which will callback and cause the front-end Logger object to access and record the value of its associated simulation object.

Although this proposal presupposes the existence of front-end Logger objects (including their ability to access and record data from the objects which they log), it does not discuss their API or design. It only concerns the design and API of the back-end components (the private Logger callback helper class as well as modifications to PropertySlot objects) and assumes that the front-end Logger objects will be designed to create, manage, and work with these back end objects.

Design

The proposed design is centered around a front-end Logging class, written in Python. These Loggers will be created by users and will be permanently associated with a single back-end loggable object; they will maintain their own logging policies as well as data storage, and will have several methods for accessing the associated value, recording a value into memory, and logging (which will combine both accessing and recording). During its creation, a Logger will register with its target object a callback class. Each Stepper that has the target back-end object in its mutator step will keep a reference to the callback, and whenever it steps, will execute that callback, which will cause the front-end Logger object to execute its log method, which will access and record the value it watches, based on its own policies.

Rationale

Logging in previous versions of the E-Cell System was based in the back-end (written entirely in C++). Although this resulted in a somewhat more cumbersome interface for creating and using Loggers from a user's perspective, the decision to use this architecture was made because of the increase in performance back-end logging gave. However, the assumption that went into this decision -- an estimation of a relatively lower ratio of time spent updating state versus time spent logging during simulation cycles-- is not the same for the models that we expect to be run using the next generation of simulation technology. Due to the rise of new algorithms such as dynamic complex species creation as well as spatial algorithms (which both increase the size of the state space, and therefore will typically raise the ratio of time updating state versus logging, as well as an increase in the complexity of the algorithms being used), we expect a much higher ratio of time spent updating state to time spent logging in the future. The conclusion is that Logging will become less performance critical within entire simulation cycles, and consequently, we propose moving logging operations to the front-end, in order to increase user ease of use.

Proposed Implementation

To implement this, two new classes will be implemented.

  • Logger (implemented in Python in the front-end)
  • LoggerInvoker (a callback class implemented in C++ in the back-end that will initiate logging by 'Logger' in the front-end)

And the PropertiedClass class will be modified so that it can store a list of LoggerInvoker pointers. Furthermore, as this new logging scheme will rely on interaction between front-end and back-end objects, the various classes involved (PropertiedClass, LoggerInvoker, and Logger) will be reflected across the front/back end using boost::python.

The python Logger will have an API that will include the following capabilities:

  • Instantiation by passing a full property name ( corresponding to __init__(self, aFullPropertyName) )
  • Getting the value of the attached property name at any time ( getValue(self) ).
  • Logging the value of the attatched property name (log(self), which will use the self.getValue function along with the global getCurrentTime to push a point into its own storage, modulo logging policies.)
  • Unregistering itself (informing the PropertiedClass to remove and delete its callback).

The LoggerInvoker class will be written in C++. It will consist of a single constructor, which will take an object* (this is a boost::python object corresponding to a class in python) that must correspond to a Logger (how to enforce this?). This pointer will be stored as the sole member data of the class. The single member function, void LoggerInvoker::initiateLogger(void), will initiate the class' stored object's log method by "mPtrObjectToInitialize->attr("log")()".

Finally, PropertiedClass will be modified to have one new piece of member data, a vector<LoggerInvoker*>, and three new member functions.

  • registerLoggerInvoker( LoggerInvoker* ) -- This function will just push_back the LoggerInvoker* sent to it.
  • deleteLoggerInvoker( LoggerInvoker* ) -- This function will check to see if the ptr is in its vector<LoggerInvoker*>. If so, it removes it and deletes the pointer.
  • log -- This function runs down the vector and calls initiateLogging() for each pointer in that list.

Comments

Comments by Koichi

  • Make clear that this proposal discusses only the backend part of the logging.
  • In Design section, the member function of of a Logger to get callback is mentioned to take a real value. This is not necessary. When it gets a callback call, the Logger already knows how to get the value.
  • Mention that the cost doing logging is becoming increasingly trivial as the scale of models and complexity of computation gets large.
  • The classname LoggerCallback; there should be a better name.
  • The function log(real); simply log()?

Comments by Moriyoshi

  • We'd better have a little more abstraction here. Logging is wholly a frontend's interest and therefore the frontend has to be made to receive only primitive messages from the backend. Events involved in changes according to the simulation are notified through an event dispatcher to the frontend in either synchronous or asynchronous manner and then some actual logger in the frontend receives the events and reports them as an log entries (to the user.)