Skip to content

Commit

Permalink
Pass up to 8.2 + some transversal fixes.
Browse files Browse the repository at this point in the history
  • Loading branch information
@jfabry
jfabry committed May 25, 2015
1 parent 64d0dc5 commit 925cc3a
Showing 1 changed file with 52 additions and 60 deletions.
112 changes: 52 additions & 60 deletions Fuel/Fuel.pillar
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,7 @@ Before going into details we present the ideas behind Fuel and it's main feature
; Object-Oriented Design
: A requirement from the onset was to have a good object-oriented design and to avoid special support from the virtual machine. In addition, Fuel has a complete test suite (over 600 unit tests), with a high degree of code coverage. Fuel code is also well-commented, both for classes and methods.

!!! Installation and demo
@Installation
!!! Installation and Demo

Fuel 1.9 is available by default in Pharo since version 2.0 of Pharo. Therefore you do not need to install it.
The ""default packages"" work out of the box in Pharo 1.1.1, 1.1.2, 1.2, 1.3, 1.4, 2.0, 3.0 and 4.0 and Squeak 4.1, 4.2, 4.3, 4.4, 4.5.
Expand Down Expand Up @@ -90,7 +89,7 @@ materializedArray third
value: 'The materialized block closure can be properly evaluated.'.
]]]

!!! Some links
!!! Some Links

- The home page of Fuel is *http://rmod.lille.inria.fr/web/software/Fuel*.
- Fuel's source code is stored at *http://smalltalkhub.com/#!/~Pharo/Fuel*.
Expand All @@ -104,7 +103,7 @@ materializedArray third
!! Getting Started
@GettingStarted

!!!Basic examples
!!!Basic Examples
Fuel offers some class-side messages to ease more common uses of serialization (==serialize:toFileNamed:==)
and materialization (==materializeFromFileNamed:==). The next example writes to and reads from a file:

Expand Down Expand Up @@ -163,7 +162,7 @@ Fuel does not care to what kind of stream it writes its data. This makes it easy
gzip close ].
]]]

!!!Showing a progress bar
!!!Showing a Progress Bar

Sometimes it is nice to see progress updates on screen. Use the message ==showProgress== in this case. The progress bar functionality is available from the
==FuelProgressUpdate== package, so load that first:
Expand Down Expand Up @@ -207,7 +206,7 @@ graph, there is no unique way of serializing it and because of this Fuel offers

@@authorToDo JF dynamic and static? which is which?

!!!Default globals
!!!Default Globals

By default, Fuel considers the following objects as globals, i.e., it will store just their name:

Expand All @@ -220,7 +219,7 @@ By default, Fuel considers the following objects as globals, i.e., it will store

@@authorToDo JF But what about the inverse of 3.2: DO export a certain global

!!!Duplication of custom globals
!!!Duplication of Custom Globals

With this following code snippet, we show that by default a global value is not serialized as a global. In such a case it is duplicated on materialization.

Expand Down Expand Up @@ -259,7 +258,8 @@ aSerializer
[ (FLMaterializer materializeFromFileNamed: 'g.fuel') == SomeGlobal ] assert.
]]]

!!!Changing the environment
!!!Changing the Environment
@sec:ManagingGlobals

The default lookup location for globals is ==Smalltalk globals==. This can be changed by using the message ==globalEnvironment:== during serialization and
materialization.
Expand Down Expand Up @@ -338,9 +338,9 @@ User >> fuelAfterMaterialization
!!!Substitution on Serialization

Sometimes it is useful to serialize something different than the original object, without altering the object itself. Fuel proposes two different ways to do
this: a dynamic way and a static way.
this: dynamically and statically.

!!!!Dynamic way
!!!!Dynamically

You can establish a specific substitution for a particular serialization. Let's illustrate with an example, where the graph includes a ==Stream== and you want
to serialize ==nil== instead.
Expand All @@ -364,9 +364,9 @@ objectToSerialize := { 'hello' . '' writeStream}.

After executing this code, ==materializedObject== will contain ==#('hello' nil)==, i.e. without the instance of a ==Stream==.

!!!! Static way
!!!! Statically
You can also do substitution for each serialization of an object by overriding its ==fuelAccept:== method. Fuel visits each object in the graph by sending this
message to determine how to trace and serialize it.
message to determine how to trace and serialize it. The argument of the message is an instance of a ==FLMapper== subclass.

As an example, imagine we want to replace an object directly with nil. In other words, we want to make all objects of a class transient, for example all
==CachedResult== instances. For that, we should implement:
Expand Down Expand Up @@ -423,7 +423,7 @@ In this case, the substituted user (i.e., the one with the empty history) will b
In the same way that we may want to customize object serialization, we may want to customize object materialization. This can be done either by treating an
object as a globally obtained reference, or by hooking into instance creation.

!!!! Global reference
!!!! Global References

Suppose we have a special instance of ==User== that represents the admin user, and it is a unique instance in the image. In the case that the admin user is
referenced in our graph, we want to get that object from a global when the graph is materialized. This can be achieved by modifying the ""serialization""
Expand All @@ -443,7 +443,7 @@ User >> fuelAccept: aGeneralMapper
During serialization the admin user won't be serialized but instead its global name and selector are stored. Then, at materialization time, Fuel will send the
message ==admin== to the class ==User==, and use the returned value as the admin user of the materialized graph.

!!!! Hooking into instance creation
!!!! Hooking Into Instance Creation

Fuel provides two hook methods to customise how instances are created: ==fuelNew== and ==fuelNew:==.

Expand Down Expand Up @@ -540,12 +540,13 @@ the class shape. Now imagine we previously serialized an instance of ==Point== a

+Example of changes to a class>file://figures/ClassChanges.png|width=70|label=figClassChanges+

@@authorToDo Fig reference is broken.

Let's start with the simple cases. If a variable was ""inserted"", its value will be ==nil==. If ""removed"", it is also obvious: the serialized value will be
ignored. ""Change of Order"" of instance variables is handled by Fuel automatically.
Let's start with the simple cases. If a variable was ""inserted"", its value will be ==nil==. If it was ""removed"", it is also obvious: the serialized value will be
ignored. The ""change of Order"" of instance variables is handled by Fuel automatically.

A more interesting case is when a variable was ""renamed"". Fuel cannot automatically guess the new name of a variable, so the change will be understood by Fuel
as two independent operations: an insertion and a removal. To resolve this problem, the user can tell the Fuel materializer which are the renamed variables by
as two independent operations: an insertion and a removal. To resolve this problem, the user can tell the Fuel materializer which variables are renamed by
using the message ==migratedClassNamed:variables:==. It takes as first argument the name of the class and as second argument a mapping from old names to new
names. This is illustrated in the following example:

Expand All @@ -566,18 +567,19 @@ FLMaterializer newDefault

Lastly, Fuel defines the message ==migrateClassNamed:toClass:variables:== that combines both ""class and variable rename"".

Additionally, the method ==globalEnvironment:==, showed in Section *@ManagingGlobals*, is useful for migrations: you can prepare an ad-hoc environment
dictionary with the same keys that were used during serialization, but with the new classes as values.
Additionally, the method ==globalEnvironment:==, shown in Section *@sec:ManagingGlobals*, is useful for migration of global variables: you can prepare an ad-hoc environment dictionary with the same keys that were used during serialization, but with the new classes as values.

@@note A class could also change its ""layout"". For example, Point could change from being ""fixed"" to ""variable"". Layout changes from fixed to variable format are automatically handled by Fuel. Unfortunately, the inverse (variable to fixed) is not supported so far.
@@authorToDo reference is broken?

@@note A class could also change its ""layout"". For example, Point could change from being ""fixed"" to ""variable"". Layout changes from fixed to variable format are automatically handled by Fuel. Unfortunately, the inverse (variable to fixed) is not supported yet.


%=========================================================================%

!! Fuel Format Migration

Until now, each Fuel version has its own stream format. Furthermore, each version is ""not"" compatible with the others. This means that when upgrading Fuel, we
will need to convert our serialized streams.
Until now, each Fuel version has used its own stream format, which is ""not"" compatible with the format of other versions. This means that when upgrading Fuel, we
will need to convert our serialized streams. This is done by using the old version of Fuel to materialize a stream, keeping a reference to this object graph, and then loading the new version of Fuel and serializing the object graph back to a file.

We include below an example of such a format migration. Let's say we have some files serialized with Fuel 1.7 in a Pharo 1.4 image and we want to migrate them
to Fuel 1.9.
Expand All @@ -591,6 +593,7 @@ fileNames := #('a.fuel' 'b.fuel' 'c.fuel' 'd.fuel' 'e.fuel').
objectsByFileName := Dictionary new.

(ConfigurationOfFuel project version: oldVersion) load.
"Need to do it like this otherwise the class is decided at compile time."
materializerClass := Smalltalk at: #FLMaterializer.

fileNames do: [ :fileName |
Expand All @@ -599,6 +602,7 @@ fileNames do: [ :fileName |
put: (materializerClass materializeFromFileNamed: fileName) ].

(ConfigurationOfFuel project version: newVersion) load.
"Need to do it like this otherwise the class is decided at compile time."
serializerClass := Smalltalk at: #FLSerializer.

objectsByFileName keysAndValuesDo: [ :fileName :objects |
Expand All @@ -607,38 +611,42 @@ objectsByFileName keysAndValuesDo: [ :fileName :objects |
toFileNamed: 'migrated-', fileName ].
]]]

@@note Note 1: We assume in this example that the number of objects to migrate can be materialized all together at the same time. This assumption may be wrong. In such case, you could adapt the script to split the list of files and do the migration in parts.

@@note Note 2: It is necessary to fetch the classes in the System Dictionary after the desired Fuel version has been loaded.
We assume in this example that the number of objects to migrate can be materialized all together at the same time. This assumption may be wrong. In such case, you could adapt the script to split the list of files and do the migration in parts.

@@note Note 3: This script should be evaluated in the original image. For example, we don't guarantee that Fuel 1.7 loads in Pharo 2.0, but we know that Fuel 1.9 loads in Pharo 1.4.
@@note This script should be evaluated in the original image. We don't guarantee that Fuel 1.7 loads in Pharo 2.0, but we do know that Fuel 1.9 loads in Pharo 1.4.

%=========================================================================%

!! Debugging
!! Fuel Development: Debugging Tools

There are a couple of packages that help us to debug Fuel, and they are loaded as follows:

[[[language=Smalltalk
Gofer it
url: 'http://smalltalkhub.com/mc/Pharo/Fuel/main';
package: 'ConfigurationOfFuel';
load.

(ConfigurationOfFuel project version: #stable)
load: #(FuelDebug FuelPreview).
]]]

There are a couple of packages that help us debugging Fuel. To understand the output of the tools, you should know some basics of how Fuel works internally.
We now first talk about the internals of Fuel before we present the different tools: a graph viewer, a serialization logger and a materialization logger.

!!!Serialization
The most important thing to know is that serialization is split in two main steps: analysis and encoding.
!!!Internal Workings

!!!!Analysis
The analysis phase consists of walking the graph from the specified root object and mapping each traversed object to its corresponding groupi, called a
""cluster"".
To understand the output of these tools, you should first know some basics of how Fuel works internally, both for serialization and materialization.

!!!!Encoding
After analysis, we write the graph to the stream linarly, in these steps:
;Serialization
:The most important thing to know is that serialization is split in two main steps: analysis and encoding. The analysis phase consists of walking the graph from the specified root object and mapping each traversed object to its corresponding group, called a ""cluster"". The encoding phase runs after analysis, and it writes the graph to the stream linarly, in these steps:

#header
#for each cluster, instances part
#for each cluster, references part
#trailer

!!!Materialization
Because of the extra effort put into serialization, we can materialize the graph in a single phase.

!!!!Decoding
We decode the graph by reading the input stream linearly, in the same order it was written. The materialization steps are obviously analogous to the ones above:
;Materialization
: Because of the extra effort put into serialization, we can materialize the graph in a single phase. We decode the graph by reading the input stream linearly, in the same order it was written. The materialization steps are obviously analogous to the ones above:

#header
#for each cluster, instances part
Expand All @@ -649,24 +657,8 @@ It is important to understand that references are ""not"" stored together with t
stored together, after the references. We use this to materialize all the references in a single step, when we know that all the objects have already been
materialized.

!!!Debug Tools
Ensure you have them with:

[[[language=Smalltalk
Gofer it
url: 'http://smalltalkhub.com/mc/Pharo/Fuel/main';
package: 'ConfigurationOfFuel';
load.

(ConfigurationOfFuel project version: #stable)
load: #(FuelDebug FuelPreview).
]]]

Here is a list of some useful class comments.


!!!!FLGraphViewBuilder
I add draw capabilities to analysis in ==FuelDebug== package.
!!!FLGraphViewBuilder
This class adds drawing capabilities to the analysis in the ==FuelDebug== package.

Right-click a node to inspect it. Some examples:

Expand Down Expand Up @@ -695,7 +687,7 @@ Figure *@figFuelPreview* shows how they look like.
+Visual preview of graph to be serialized>file://figures/FuelPreview.png|width=60|label=figFuelPreview+
_

!!!!FLDebugSerialization
!!!FLDebugSerialization
I am a serialization which facilitates debugging, by logging the stream position before and after main steps of ==FLSerialization==, including cluster
information. Obviously, you should be familiar with the algorithm to understand the output log.

Expand All @@ -714,7 +706,7 @@ Then, inspect the output log:
FLDebugSerialization last log.
]]]

!!!!FLDebugMaterialization
!!!FLDebugMaterialization
I am a materialization which facilitates debugging, by logging the stream position before and after main steps of ==FLMaterialization==, including cluster
information. Obviously, you should be familiar with the algorithm to understand the output log.

Expand Down

0 comments on commit 925cc3a

Please sign in to comment.