Components in the SystemML Namespace are subject to a simple Version Control system. Each Component is tagged with a "Release" number, which is incremented with each update to the Component. The two aspects of the Component that are involved in Version Control are the Component Specification, and the Component Implementation.
How things should work is as follows. Release Zero should include full documentation of the Component Specification along with the first Component Implementation. Later releases may not revoke aspects of that specification (particularly, the interface provided by the Component), but they may extend it. However, implementation being what it is, there is a risk that the first implementation (Release Zero) does not, in fact, meet the Specification. In addition, documentation being what it is, there is a risk that the initial specification will be incomplete.
One of SystemML's main aims is persistence - a System executed today will produce identical results if run in ten years from now. The Version Control strategy of SystemML offers compliance with this directive in the presence of the risks described above. It is detailed below.
Component implementations may be updated, or new implementations (for instance, in different languages) may be added to the tree. Each update requires a new release, and a release number increment. We list the possible types of update, below, and outline the strategy that the Server and a Client (e.g. BRAHMS) must follow to ensure that the risks are managed effectively.
The Release number of every Component Implementation used in an Execution must be recorded by the Execution Client in the Execution Report, such that identical results can be regenerated at a later date, even if later release have changed the Component's behaviour, just from the output files. This provides the last line of defence against unintended behaviour changes through auditing - if uncertainties arise, the original implementation files can be used to reproduce an earlier Execution exactly. If necessary, the behaviour changes in later releases can then be assessed against this reference, and the System brought up to date if desired. However, the update strategy below will greatly mitigate the risk of such unintended behaviour changes.
The general approach is to assign a status to each Release, which can be changed in future. The status of a release helps the Client to decide how to use the Release, and to inform the user of what the effects might be. Clients should maintain a record of which releases are in use to run any particular System, and manage updating these configurations carefully, as described below.
It is strongly recommended that the Client offer validation services during upgrades to new Releases. This can be achieved by executing the System that has been sent for Execution twice, once with the older release and once with the new. Hash checking of the output files will reveal any change in behaviour. Failing this, the user should be advised to look out for behaviour changes, and be given the opportunity to observe behaviour before and after the upgrade.
If behaviour changes are uncovered, the Client (or the User) should notify the server, and submit a test suite that proves the behaviour change. The server will confirm the behaviour change, and then change the release status to "revoked". Clients will roll back to the previous release on their next update from the server, and Clients that have not yet updated will ignore the revoked release.
It will then be up to the Component maintainer to trace the behaviour anomaly (using the submitted test suite) and correct it. A new release can then be made, and the update process proceeds as before. The maintainer (or any User) can update test suites for any Component at any time, to provide the Server with additional tools to automatically identify behaviour changes.
Some reasons for new Releases
Component Specification, as noted above, can not be revoked after publication. This is to ensure that older code that interacts with the Component class will still be able to interact correctly with newer releases. However, later releases may extend the Specification, i.e. the interface. If aspects of the interface need to be revoked, or if previously-specified behaviour has to change, a new Class must be generated. This is not negotiable, and compliance is enforced by the infrastructure.
A new release with an extended specification initially has the status "unchanged", and is flagged as an "extension". When the Client finds such a new release, it should offer the user the opportunity to update all uses of the Component to the new Release, and advise that behaviour changes should not occur, and also direct the user to the Release Notes that detail the extension. Validation should be applied.
Later releases can change internal operation with respect to earlier releases, so long as this does not violate their previously-published interfaces and specification. This will usually mean performance improvements (code optimisation) or fixes of bugs that cause operation to fail completely under certain conditions.
These updates can be safely applied as for extensions, so releases initially have the status "unchanged", and are flagged as "fix" or "performance". Clients can generally update to these new releases with little interaction with the user, but at the very least notification should be given, and the user should be directed to the Release Notes. Validation should be applied.
Some types of bug will inevitably emerge that produce non-failing but incorrect (not meeting the specification) behaviour under certain conditions. These bugs require changes to previously-released behaviour (but not to the specification), and are thus of a different class to bugs that cause complete failure. That is, they can lead to behaviour changes in existing Systems - even if those behaviour changes make the behaviour more "correct", they still constitute non-compliance with the aim of persistence.
Whilst such bugs could be fixed in a new Class, rather than a new Release, this discards the opportunity to correct the behaviour of existing Systems. Therefore, we use a compromise strategy for such releases. This compromise allows the repair of old Systems, without the risk of introducing unexpected behaviour changes, and with new Systems having corrected behaviour.
The strategy is to make a new release, and mark its status as "changed". Documentation should be included with the release to indicate why the update is being made, and exactly what is expected to change. Clients should not automatically start using these new releases to run existing Systems, but can safely use them in newly-authored Systems (though the user should be notified, as for any update). It is up to the Client exactly how update of older Systems is negotiated with the User. They can be updated to use the new release, but the user should confirm (perhaps using validation tools provided by the Client) that behaviour of their particular System is unchanged, or changed only in a negligible way, by the update.
Since a currently standard SystemML System File contains no implementation information, it does not indicate which release of any particular Component should be used. A standard Client Execution Report File should, however, indicate exactly which implementations were used during the Execution. The list of implementations used (and it is a flat list) is referred to as an Implementation Configuration. This data can be stored in the SystemML file, if this is the format the Client works with natively, through the
It is up to the client how Implementation Configuration is managed, but a sensible approach is outlined here, for illustration. The latest set of updates is taken to define the "current" Implementation Configuration, at all times. When a System is authored, the current Implementation Configuration in the environment should be stored as part of the System. The System is, thus, locked into that configuration. On subsequent executions, if updates have occurred and the current configuration has changed, the User can be offered the chance to update to the latest available configuration (passing through a client-provided validation step), or not, as desired.
Downgrades and Language Switches
The risks and strategy for downgrades (moving to an earlier Release) and switching to a different language of implementation (e.g. for performance reasons) are just the same as for upgrades within a series of Releases for a single language. These should be handled in the same way.
|This is a documentation page for the BRAHMS Modular Execution (Simulation) Framework. For details of licensing and distribution, please click here. To advise of an error or omission, please click here.|