diff --git a/NeoCSV/NeoCSV.pier b/NeoCSV/NeoCSV.pier new file mode 100644 index 0000000..725c84f --- /dev/null +++ b/NeoCSV/NeoCSV.pier @@ -0,0 +1,207 @@ +! Handling CSV with NeoCSV + +CSV (Comma Separated Values) is a popular data-interchange format. NeoCSV is an elegant and efficient standalone Pharo framework to read and write CSV converting to or from Pharo objects. This chapter has been written by +Sven Van Caekenberghe the author of NeoCSV and many other nicely designed Pharo libraries. + +!! An introduction to CSV + +CSV is a lightweight text-based de facto standard for human-readable tabular data interchange. Essentially, the key characteristics are that CSV (or more generally, delimiter separated text data): + +- is text-based (ASCII, Latin1, Unicode), +- consists of records, 1 per line (any line ending convention), +- where records consist of fields separated by a delimiter (comma, tab, semicolon), +- where every record has the same number of fields, and +- where fields can be quoted should they contain separators or line endings. + +Here are some relevant links: + +*http://en.wikipedia.org/wiki/Comma-separated_values* +*http://tools.ietf.org/html/rfc4180* + +Note that there is not one single official standard specification. + +!! NeoCSV + +The NeoCSV framework contains a reader (==NeoCSVReader==) and a writer (==NeoCSVWriter==) to parse respectively generate delimiter separated text data to or from Smalltalk objects. The goals of this project are: + +- to be standalone (have no dependencies and have little requirements), +- to be small, elegant and understandable, +- to be efficient (both in time and space), and +- to be flexible and non-intrusive. + +To use either the reader or the writer, you instantiate them on a character stream and use standard stream access messages. Here are two examples: + +The first one read a sequences of data separated by ==,== and containing some line breaks. The reader produces arrays corresponding to the lines with the data. + +[[[ +(NeoCSVReader on: '1,2,3\4,5,6\7,8,9' withCRs readStream) upToEnd. + --> #(#('1' '2' '3') #('4' '5' '6') #('7' '8' '9')) +]]] + + +The second proceeds from the inverse: given a set of data as arrays it produces comma separated lines. + +[[[ +String streamContents: [ :stream | + (NeoCSVWriter on: stream) + nextPutAll: #( (x y z) (10 20 30) (40 50 60) (70 80 90) ) ]. + + --> + '"x","y","z" +"10","20","30" +"40","50","60" +"70","80","90" +' +]]] + +!! Generic Mode + +NeoCSV can operate in generic mode without any further customization. While writing, + +- record objects should respond to the ==do:== protocol, +- fields are always sent ==asString== and quoted, and +- CRLF line ending is used. + +While reading, + +- records become arrays, +- fields remain strings, +- any line ending is accepted, and +- both quoted and unquoted fields are allowed. + +The standard delimiter is a comma character. Quoting is always done using a double quote character. A double quote character inside a field will be escaped by repeating it. Field separators and line endings are allowed inside a quoted field. Any whitespace is significant. + + +!! Customizing NeoCSVWriter + +Any character can be used as field separator, for example: + +[[[ +neoCSVWriter separator: Character tab +]]] +or +[[[ +neoCSVWriter separator: $; +]]] + + +Likewise, any of the three common line end conventions can be set: in the following example we set carriage return. + +[[[ +neoCSVWriter lineEndConvention: #cr +]]] + +There are 3 ways a field can be written (in increasing order of efficiency): + +- quoted - converting it with ==asString== and quoting it (the default), +- raw - converting it with ==asString== but not quoting it, and +- object - not quoting it and using ==printOn:== directly on the output stream. + +Obviously, when disabling quoting, you have to be sure your values do not contain embedded separators or line endings. If you are writing arrays of numbers for example, this would be the fastest way to do it: + +[[[ +neoCSVWriter + fieldWriter: #object; + nextPutAll: #( (100 200 300) (400 500 600) (700 800 900) ) +]]] + +The ==fieldWriter== option applies to all fields. + +If your data is in the form of regular domain level objects it would be wasteful to convert them to arrays just for writing them as CSV. NeoCSV has a non-intrusive option to map your domain object's fields: You add field specifications based on accessors. This is how you would write an array of Points. + +[[[ +neoCSVWriter + nextPut: #(x y); + addFields: #(x y); + nextPutAll: { 1@2. 3@4. 5@6 } +]]] + +Note how we first write the header (before customizing the writer). The ==addField:== and ==addFields:== methods arrange for the specified accessors to be performed on the incoming objects to produce values that will be written by the fieldWriter. Additionally, there is a protocol to specify different field writing behavior per field, using ==addQuotedField:==, ==addRawField:== and ==addObjectField:==. To specify different field writers for an array (actually an SequenceableCollection subclass), you can use the methods first, second, third, ... as accessors. + +SD: What is the difference between nextPut and addFields: +SD: for the comics should I add a method to handle the relationship to other domain objects? + +!! Customizing NeoCSVReader + +The parser is flexible and forgiving. Any line ending will do, quoted and non-quoted fields are allowed. + +Any character can be used as field separator, for example: + +[[[ +neoCSVReader separator: Character tab +]]] +or + +[[[ +neoCSVReader separator: $; +]]] + +NeoCSVReader will produce records that are instances of its ==recordClass==, which defaults to ==Array==. All fields are always read as Strings. If you want, you can specify converters for each field, to convert them to integers or floats, any other object. Here is an example: + +[[[ +neoCSVReader + addIntegerField; + addFloatField; + addField; + addFieldConverter: [ :string | Date fromString: string ]; + upToEnd. +]]] + +Here we specify 4 fields: an integer, a float, a string and a date field. Field conversions specified this way only work on indexable record classes, like Array. + +In many cases you will probably want your data to be returned as one of your domain objects. It would be wasteful to first create arrays and then convert all those. NeoCSV has non-intrusive options to create instances of your own object classes and to convert and set fields on them directly. This is done by specifying accessors and converters. Here is an example for reading Associations of Floats. + + +[[[ +(NeoCSVReader on: '1.5,2.2\4.5,6\7.8,9.1' withCRs readStream) + recordClass: Association; + addFloatField: #key: ; + addFloatField: #value: ; + upToEnd. +]]] + +For each field you give the mutator accessor to use, as well as an implicit or explicit conversion block. + +One thing that it will enforce is that all records have an equal number of fields. When there are no field accessors or converters, the field count will be set automatically after the first record is read. If you want you could set it upfront. When there are field accessors or converters, the field count will be set to the number of specified fields. + +!! Reading a lot of objects + +Handling large CSV file is possible with NeoCVS since it pays attention to have +For example its ==do:== is implemented as streaming over the record one by one, never holding more than one in memory. +Here is a little example generating first over 300 Mb of data and loading it partly. + +[[[ +'paul.csv' asFileReference writeStreamDo: [ :file| + ZnBufferedWriteStream on: file do: [ :out | + (NeoCSVWriter on: out) in: [ :writer | + writer writeHeader: { #Number. #Color. #Integer. #Boolean}. + 1 to: 1e7 do: [ :each | + writer nextPut: { each. #(Red Green Blue) atRandom. 1e6 atRandom. #(true false) atRandom } ] ] ] ]. +]]] + + +This results in a 300Mb file: + +[[[ +]]]$ ls -lah paul.csv +-rw-r--r--@ 1 sven staff 327M Nov 14 20:45 paul.csv +$ wc paul.csv + 10000001 10000001 342781577 paul.csv +]]] + + +This is a selective read and collect (loads about 10K records): + +[[[ +Array streamContents: [ :out | + 'paul.csv' asFileReference readStreamDo: [ :in | + (NeoCSVReader on: (ZnBufferedReadStream on: in)) in: [ :reader | + reader skipHeader; addIntegerField; addSymbolField; addIntegerField; addFieldConverter: [ :x | x = #true ]. + reader do: [ :each | each third < 1000 ifTrue: [ out nextPut: each ] ] ] ] ]. +]]] + + +!! Conclusion + +NeoCSV provides a good support to emit or import comma separated data. + \ No newline at end of file diff --git a/NeoJSON/NeoJSON.pier b/NeoJSON/NeoJSON.pier new file mode 100644 index 0000000..f374568 --- /dev/null +++ b/NeoJSON/NeoJSON.pier @@ -0,0 +1,226 @@ +! JSON + +JSON (JavaScript Object Notation) is a popular data-interchange format. NeoJSON is an elegant and efficient standalone Smalltalk framework to read and write JSON converting to or from Smalltalk objects developed and actively maintained by Sven Van Caekenberghe. + + + + + +!!An introduction to JSON + +JSON is a lightweight text-based open standard designed for human-readable data interchange. It was derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for many languages. + +Here are some relevant links: *http://www.json.org/*, *http://en.wikipedia.org/wiki/Json* and *http://www.ietf.org/rfc/rfc4627.txt?number=4627*. + +There are only a couple of primitive types in JSON: + +-numbers (integer or floating point) +-strings +-the boolean constants ==true== and ==false== +- ==null== + +Only two composite types exist: + +-lists (an ordered sequenece of values) +-maps (an unordered associative array, mapping string property names to values) + +That is really all there is to it. No options or additions are defined in the standard. + +!! NeoJSON + +The NeoJSON framework contains a reader (==NeoJSONReader==) and a writer (==NeoJSONWriter==) to parse respectively generate JSON to or from Pharo objects. The goals of this project are: + +-to be standalone (have no dependencies and have little requirements) +-to be small, elegant and understandable +-to be efficient (both in time and space) +-to be flexible and non-intrusive + +Compared to other Smalltalk JSON frameworks, NeoJSON has + +-less dependencies and little requirements +-can be more efficient (be faster and use less memory) +-allows for the use of schemas and mappings + +!!Primitives + +Obviously, the primitive types are mapped to corresponding Pharo classes. While reading: + +- numbers become Integers or Floats +- strings become Strings +- booleans become Booleans +- ==null== become ==nil== + +While writing + +- Numbers are converted to floats, except for Integers that become integers +- Strings and subclasses become strings +- Booleans become booleans +- ==nil== becomes ==null== + + +!! Generic Mode + +NeoJSON can operate in a generic mode that requires no further configuration. + +!!! Reading from JSON + +While reading: + +- maps become instances of mapClass, ==Dictionary== by default +-lists become instance of listClass, ==Array== by default + +These are some examples reading in generic mode: + +[[[NeoJSONReader fromString: ' [ 1,2,3 ] '. + +NeoJSONReader fromString: ' [ 3.14159, true, false, null, "string" ] '. + +NeoJSONReader fromString: ' { "x" : 1, "y" : 2 } '.]]] + + +The reader can be customized to use a different mapClass or listClass. There is also an option to convert all map keys to symbols, which is off by default. + +SD: example of how to specify options + + + +!!!Writing to JSON + +While writing: + +-Dictionary and SmallDictionary become maps +-all other Collection classes become lists +-all other Objects are rejected + +Here are some examples writing in generic mode: + +[[[ +NeoJSONWriter toString: #(1 2 3). + +NeoJSONWriter toString: { Float pi. true. false. 'string' }. + +NeoJSONWriter toStringPretty: (Dictionary new at: #x put: 1; at: #y put: 2; yourself). +]]] +NeoJSONWriter can output either in a compact format (the default) or in a pretty printed format. + +SD: examples how to do that. + + +In order to use the generic mode, you have to convert your domain objects to and from Dictionaries and SequenceableCollections. This is relatively easy but not very efficient, depending on the use case. + + +!! Schemas and Mappings + +NeoJSON allows for the optional specification of schemas and mappings to be used when writing and/or when reading. A ==NeoJSONMapper== holds a number of schemas. Each schema is identified by either a class or a symbol. Each schema specifies a mapping, an object that will help in doing the actual reading or writing. + +The most common mapping deals with objects that define a number of named properties or attributes. These can be defined based on instance variables (optionally derived by reflection), accessors (getter/setter pairs) or even blocks. Such an object mapping is identified by a Smalltalk class, which is also used to create new instances. Each property mapping can have an optional value schema to be used recursively when reading and/or writing property values. + +The less common custom mapping holds a generic reader and/or writer block to deal with special cases such as specific collection types with an optional schema for the elements, or a direct mapping of semi primitive types such as Date or DateAndTime. + +A mapping can be specified explicitely on a mapper, or can be resolved using the #neoJsonMapping: class method. + +Here are some examples of mappings: + +[[[ +mapper mapAllInstVarsFor: Point. + +mapper for: TestObject do: [ :mapping | + mapping mapInstVars: #(id name). + (mapping mapInstVar: #timestamp to: 'created-at') valueSchema: DateAndTime. + (mapping mapInstVar: #points) valueSchema: #ArrayOfPoints. + (mapping mapInstVar: #bytes) valueSchema: ByteArray ]. + +mapper for: DateAndTime customDo: [ :mapping | + mapping decoder: [ :string | DateAndTime fromString: string ]. + mapping encoder: [ :dateAndTime | dateAndTime printString ] ]. + +mapper for: #ArrayOfPoints customDo: [ :mapping | + mapping listOfElementSchema: Point ]. + +mapper for: #DictionaryOfPoints customDo: [ :mapping | + mapping mapWithValueSchema: Point ]. + +mapper for: ByteArray customDo: [ :mapping | + mapping listOfType: ByteArray ] +The classes NeoJSONReader and NeoJSONWriter are subclasses of NeoJSONMapper. When writing, mappings are used when arbitrary objects are seen. For example, in order to be able to write an array of points, you could do as follows: + +String streamContents: [ :stream | + (NeoJSONWriter on: stream) + prettyPrint: true; + mapInstVarsFor: Point; + nextPut: (Array with: 1@3 with: -1@3) ]. + +]]] + +Collections are handled automatically, like in the generic case. When reading, a mapping is used as a binding or an explicit type specifying what Pharo objects that you want to read. Here is a very simple case, reading a map as a point: + + +[[[ +(NeoJSONReader on: ' { "x" : 1, "y" : 2 } ' readStream) + mapInstVarsFor: Point; + nextAs: Point. +]]] + +Since JSON lacks a universal way to specify the class of an object/map, we have to specify the target schema that we want to use as an argument to ==nextAs:==. + +With custom mappings, it is possible to + +-define the schema of the elements of a list +-define the schema of the elements of a list as well as the class of the list +-define the schema of the values of a map In fact, NeoJSONCustomMapping can be extended to implement even more specialized mappings. + +Finally, here is a more complex example, reading a list of maps as an array of points: + +[[[ +(NeoJSONReader on: '[ { "x" : 1, "y" : 2 }, { "x" : 3, "y" : 4 } ]' readStream) + mapInstVarsFor: Point; + for: #ArrayOfPoints customDo: [ :mapping | + mapping listOfElementSchema: Point ]; + nextAs: #ArrayOfPoints. +]]] + +NeoJSON deals efficiently with mappings: the minimal amount of intermediary structures are created, which is quite different from the generic case. + +!!Internals + +On modern hardware, NeoJSON can write or read in the tens of thousands of small objects per second. Several benchmarks are included in the unit tests package. + + + +!!Emitting null values + +When mapping my objects with NeoJSONWriter, it is not writing the properties whose values are null. + +[[[ + NeoJSONPropertyMapping>>#writeObject: anObject on: jsonMapWriter + | value | + value := getter value: anObject. + value + ifNotNil: [ jsonMapWriter writeKey: propertyName value: value as: valueSchema ] +]]] + +It is a good default (saves space, and cpu cycles). But in there are cases where the consumer of the JSON objects expect the objects to have all the attributes previously defined. + +NeoJSON is cool and it supports this behavior. Set the writeNil: to true as follows and you are done. + +[[[ + + String streamContents: [ :stream | + (NeoJSONWriter on: stream) + writeNil: true; + mapAllInstVarsFor: Point; + nextPut: Point new. +]]] +which will give you: +[[[ + {"x":null,"y":null} +]]] +instead of: + +[[[ + {} +]]] + +!! Conclusion + +NeoJSON is a powerful library to convert objects. Now Sven developed STON (Smalltalk object notation) which is closer to Pharo syntax and handles cycles and references between serialized objects. diff --git a/RenoirST/RenoirST.pier b/RenoirST/RenoirST.pier new file mode 100644 index 0000000..7c9b961 --- /dev/null +++ b/RenoirST/RenoirST.pier @@ -0,0 +1,676 @@ +! Cascading Style Sheet with RenoirSt + +RenoirST is a DSL enabling programmatic cascading style sheet generation for Pharo Smalltalk developed by Gabriel Omar Cotelli. + +The goals of RenoirST are to improve CSS integration with existing web frameworks +and to be able to write and refactor code in Pharo and deploy to CSS. Renoir features are: +common properties declaration, CSS3 selector support, important rules and media queries support. +In this tutorial we will present the key features of RenoirSt with a large set of examples. +This tutorial assumes some knowledge of CSS and Pharo Smalltalk. For a little introduction about CSS read the chapter *http://book.seaside.st/book/fundamentals/css*. + +!! Getting started + +To load the library you can load it in your 3.0 image evaluating: + +[[[ +Gofer it + url: 'http://smalltalkhub.com/mc/gcotelli/RenoirSt/main'; + configurationOf: 'RenoirSt'; + loadStable +]]] + +or download a ready to use image from the Contribution CI Server *https://ci.inria.fr/pharo-contribution/job/RenoirSt/*. + + +!! Introduction + + +The main entry point for the library is the class ==CascadingStyleSheetBuilder==. Let's see some minimalist example. Copy in a workspace and Inspect-it (Alt\+i): + +[[[ +CascadingStyleSheetBuilder new build +]]] + +Beautiful! you have now an inspector on your first (empty and useless) style sheet. Let's do now something more useful. Real stylesheets are composed of rules (or rule-sets), where each one has a selector and a declaration group. The selector determines if the rule applies to some element in the DOM, and the declaration group specifies the style to apply. + +!! Basics + +Our first style sheet will simply assign a margin to every div element in the DOM. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | style margin: 2 px ]; + build +]]] + +the expected result is: + +[[[ +div +{ + margin: 2px; +} +]]] + +Let's analyze it. The message ==declareRuleSetFor:with:== is used to configure a rule-set in the builder. It uses two closures, the first one is used to define the selector and the second one the style. The selector argument of the first closure provides an API to construct the selector (more on this later). The style argument on the second closure provides the API to declare CSS properties and his values. + +The properties API is mostly defined following this rules: + +- Properties without dashes in the name are directly mapped: margin became margin: message send. +- Properties with one or more dashes are mapped using camel case: margin-top became marginTop: message send. + + +!! Basic CSS Types + +!!! Lengths + +Another interest thing is 2 px message send. This message send produces a ==CssLength==. The library provides out-of-the-box support for the length units in the CSS spec. There's extensions to Integer and Float classes allowing to obtain lengths. The supported units are: + +- em relative to font size +- ex relative to "x" height +- cm centimeters +- mm millimeteres +- in inches +- pc picas +- pt points +- px pixels (note that CSS has some special definition for pixel) + +It also supports the creation of percentages: 50 percent is expressed as ==50%== in the resulting CSS. + +Some properties requires integer or floating point values. In this cases just use the Pharo provided integer and float support. For example: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | style zIndex: 2 ]; + build +]]] + +!!! Colors + +The library also supports abstractions for properties requiring color values. It provides a shared pool ==CssSVGColors== providing easy access to colors in the SVG 1.0 list, and some abstractions (==CssRGBColor== and ==CssHSLColor==) to create colors in the rgb or hsl space including alpha support. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | + style + backgroundColor: CssSVGColors aliceBlue; + borderColor: (CssRGBColor red: 0 green: 128 blue: 0 alpha: 0.5)]; + build +]]] + +creates a style sheet for + +[[[ +div +{ + background-color: aliceblue; + border-color: rgba(0,128,0,0.5); +} +]]] + +Hint: In a real scenario don't harcode the colors like this example, put it in some object representing a theme or something similar and use a more functional name. + + +RGB-Colors also support percentage-based values: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | style borderColor: (CssRGBColor red: 0 percent green: 50 percent blue: 0 percent) ]; + build +]]] + +Notice the difference in the function name because there's no alpha channel specification: + +[[[ +div +{ + border-color: rgb(0%,50%,0%); +} +]]] + +!!! Constants + +A lot of properties values are just keyword constants. This support is provided by the class CssConstants. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | style textAlign: CssConstants justify ]; + build +]]] + +[[[ +div +{ + text-align: justify; +} +]]] + + +!! Multiple Property Values + +Some properties support a wide range of values. For example the margin property can have 1, 2 , 3 or 4 values specified. If only one value needs to be specified just provide it, in other case use an ==Array== like this: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ] + with: [:style | style margin: { 2 px. 4 px } ]; + build +]]] + +being the resulting style sheet: + + +[[[ +div +{ + margin: 2px 4px; +} +]]] + +!!URLs + +ZnUrl instances can be used as the value for properties requiring an URI. Both relative or absolute URLs are acceptable. + + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div class: 'logo' ] + with: [:style | style backgroundImage: 'images/logo.png' asZnUrl ]; + declareRuleSetFor: [:selector | selector div class: 'logo' ] + with: [:style | style backgroundImage: 'http://www.example.com/images/logo.png' asZnUrl ]; + build +]]] + +being the resulting style sheet: + +[[[ +div.logo +{ + background-image: url("images/logo.png"); +} + +div.logo +{ + background-image: url("http://www.example.com/images/logo.png"); +} +]]] + + + +!! Selectors + +So far our focus was on the style part of the rule. Let's focus now on the available selectors. Remember that a CSS selector represents a structure used to match elements in the document tree. This chapter assumes some familiarity with the CSS selectors and will not go in detail about the exact meaning of each one. For more details you can take a look at *http://www.w3.org/TR/css3-selectors/* + +!!! Type selectors + +These selectors match a specific element type in the DOM. The library provides out-of-the-box support for HTML elements, mostly using the same names that the Seaside framework. One example is the div selector used in the previous chapter. Another is the following: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector orderedList ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] + +produces + +[[[ +ol +{ + list-style-type: lower-roman; +} +]]] + +!! Combinators + +One of the most common use cases is the descendant combinator. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div orderedList ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] + +produces + +[[[ +div ol +{ + list-style-type: lower-roman; +} +]]] + + +!!! Child Combinator + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div > selector orderedList ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] + +produces + +[[[ +div > ol +{ + list-style-type: lower-roman; +} +]]] + +!! Adjacent and General Siblings Combinators + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div + selector orderedList ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] + +produces + +[[[ +div + ol +{ + list-style-type: lower-roman; +} +]]] + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector div ~ selector orderedList ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] +produces + +[[[ +div ~ ol +{ + list-style-type: lower-roman; +} +]]] + +!!Class and Id Selectors + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | (selector div class: 'pastoral') id: #account5 ] + with: [:style | style listStyleType: CssConstants lowerRoman ]; + build +]]] + +produces + +[[[ +div.pastoral#account5 +{ + list-style-type: lower-roman; +} +]]] + +!!!!Hint: +You should not hardcode the classes and ids, they should be obtained from the same object that holds them for the HTML generation. You probably have some code setting the class(es) and/or id(s) to a particular HTML element. + +!! Pseudo-Classes + +The pseudo-class concept is introduced to allow selection based on information that lies outside of the document tree or that cannot be expressed using the simpler selectors. Most pseudo-classes are supported just by sending one of the following messages link, visited, active, hover, focus, target, enabled, disabled or checked. + +Here is a small example that uses the pseudo-classes: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector anchor link ] + with: [:style | style color: CssSVGColors blue ]; + declareRuleSetFor: [:selector | selector anchor visited active] + with: [:style | style color: CssSVGColors green ]; + declareRuleSetFor: [:selector | selector anchor focus hover enabled] + with: [:style | style color: CssSVGColors green ]; + declareRuleSetFor: [:selector | (selector paragraph class: 'note') target disabled] + with: [:style | style color: CssSVGColors green ]; + declareRuleSetFor: [:selector | selector input checked ] + with: [:style | style color: CssSVGColors green ]; + build +]]] + +produces + +[[[ +a:link +{ + color: blue; +} + +a:visited:active +{ + color: green; +} + +a:focus:hover:enabled +{ + color: green; +} + +p.note:target:disabled +{ + color: green; +} + +input:checked +{ + color: green; +} +]]] + +!!!! Language Pseudo-class: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | (selector lang: 'es') > selector div ] + with: [:style | style quotes: { '"<"'. '">"' } ]; + build +]]] + +produces + +[[[ +:lang(es) > div +{ + quotes: "<" ">"; +} +]]] + +!!!! Structural Pseudo-classes + +These selectors allow selection based on extra information that lies in the document tree but cannot be represented by other simpler selectors nor combinators. + +Standalone text and other non-element nodes are not counted when calculating the position of an element in the list of children of its parent. When calculating the position of an element in the list of children of its parent, the index numbering starts at 1. + +!!!! Root Pseudo-class + +The ==:root== pseudo-class represents an element that is the root of the document. To build this kind of selector just send the message root to another selector: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector root ] + with: [:style | style color: CssSVGColors grey ]; + build +]]] + +!!!! Kind of nth-child Pseudo-classes + +The :nth-child(an\+b) pseudo-class notation represents an element that has an\+b\-1 siblings before it in the document tree, for any positive integer or zero value of n, and has a parent element. For values of a and b greater than zero, this effectively divides the element's children into groups of a elements (the last group taking the remainder), and selecting the bth element of each group. The a and b values must be integers (positive, negative, or zero). The index of the first child of an element is 1. + +In addition to this, ==:nth-child()== can take =='odd'== and =='even'== as arguments instead. The value =='odd'== is equivalent to 2n\+1, whereas =='even'== is equivalent to 2n. + +The library does not currently include the abstraction for this kind of formulae, but a plain string can be used, or just an integer if n is not required. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector childAt: '3n+1' ] + with: [:style | style color: CssSVGColors blue ]; + declareRuleSetFor: [:selector | selector childAt: 5 ] + with: [:style | style color: CssSVGColors blue ]; + build +]]] + +[[[ +:nth-child(3n+1) +{ + color: blue; +} + +:nth-child(5) +{ + color: blue; +} +]]] +The rest of the selectors in this category are modeled using the following messages: + +[[[ +nth-last-child() -> childFromLastAt: +nth-of-type() -> siblingOfTypeAt: +nth-last-of-type() -> siblingOfTypeFromLastAt: +first-child -> firstChild +last-child -> lastChild +first-of-type -> firstOfType +last-of-type -> lastOfType +only-child -> onlyChild +only-of-type -> onlyOfType +empty -> empty +]]] + +!! Pseudo-Elements + +Pseudo-elements create abstractions about the document tree beyond those specified by the document language. For instance, document languages do not offer mechanisms to access the first letter or first line of an element's content. Pseudo-elements allow authors to refer to this otherwise inaccessible information. Pseudo-elements may also provide authors a way to refer to content that does not exist in the source document. + +!!!! First line + +This selector describes the contents of the first formatted line of an element. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector paragraph firstLine ] + with: [:style | style textTransform: CssConstants uppercase ]; + build +]]] +produces + +[[[ +p::first-line +{ + text-transform: uppercase; +} +]]] + +!!!!First letter + +This pseudo-element represents the first letter of an element, if it is not preceded by any other content (such as images or inline tables) on its line. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector paragraph firstLetter ] + with: [:style | style fontSize: 200 percent ]; + build +]]] +produces + +[[[ +p::first-letter +{ + font-size: 200%; +} +]]] + +!!!! Before and After + +These pseudo-elements can be used to describe generated content before or after an element's content. The content property, in conjunction with these pseudo-elements, specifies what is inserted. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | (selector paragraph class: 'note') before ] + with: [:style | style content: '"Note: "' ]; + declareRuleSetFor: [:selector | (selector paragraph class: 'note') after ] + with: [:style | style content: '"[*]"' ]; + build +]]] + +produces +[[[ +p.note::before +{ + content: "Note: "; +} + +p.note::after +{ + content: "[*]"; +} +]]] + +!! Selector Groups + +A comma-separated list of selectors represents the union of all elements selected by each of the individual selectors in the list. For example, in CSS when several selectors share the same declarations, they may be grouped into a comma-separated list. + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | (selector div class: 'note') after , (selector paragraph class: 'note') before ] + with: [:style | style content: '"Note: "' ]; + build +]]] + +[[[ +div.note::after , +p.note::before +{ + content: "Note: "; +} +]]] + +!! Important Rules + +CSS attempts to create a balance of power between author and user style sheets (*http://www.w3.org/TR/CSS2/cascade.html#important-rules*). By default, rules in an author's style sheet override those in a user's style sheet. + +However, for balance, an ==!important== declaration takes precedence over a normal declaration. Both author and user style sheets may contain ==!important== declarations, and user ==!important== rules override author ==!important== rules. This CSS feature improves accessibility of documents by giving users with special requirements control over presentation. + +The library provides support for this feature by sending the ==beImportantDuring:== message to the style. Let's see an example: + +[[[ +CascadingStyleSheetBuilder new + declareRuleSetFor: [:selector | selector paragraph ] + with: [:style | + style beImportantDuring: [:importantStyle | + importantStyle + textIndent: 1 em; + fontStyle: CssConstants italic + ]. + style fontSize: 18 pt. + ]; + build +]]] +produces : + +[[[ +p +{ + text-indent: 1em !important; + font-style: italic !important; + font-size: 18pt; +} +]]] + +Note that the important properties must be created by sending the messages to the inner argument ==importantStyle== instead of the outer argument ==style==. + + +!! Media Queries + +A ==@media== rule specifies the target media types of a set of statements (see *http://www.w3.org/TR/CSS2/media.html*). The ==@media== construct allows style sheet rules that apply to various media in the same style sheet. Style rules outside of @media rules apply to all media types that the style sheet applies to. At-rules inside ==@media== are invalid in CSS2.1 (see *http://www.w3.org/TR/css3-mediaqueries/*). + + +The most basic media rule consists of specifying just a media type: + +[[[ +CascadingStyleSheetBuilder new + declare: [ :cssBuilder | + cssBuilder + declareRuleSetFor: [ :selector | selector id: #oop ] + with: [ :style | style color: CssSVGColors red ] + ] + forMediaMatching: [ :queryBuilder | queryBuilder type: CssMediaQueryConstants print ]; + build +]]] + +To use media queries in the library just send the message ==declare:forMediaMatching:== to the builder. The first closure is evaluated with an instance of a ==CascadingStyleSheetBuilder== and the second one with a builder of media queries. + +The media query builder will match any media type by default. To specify a media type just send it the message ==type:== with the corresponding media type. The class ==CssMediaQueryConstants== provides easy access to the following media types: braille, embossed, handheld, print, projection, screen, speech, tty and tv. + +The media query builder supports a variety of messages for additional conditions (called media features). Media features are used in expressions to describe requirements of the output device. + +The following media feature messages are supported: + +Accepting as argument a ==CssLength== +-width: +-minWidth: +-maxWidht: +-height: +-minHeight: +-maxHeight: +-deviceWidth: +-minDeviceWidth: +-maxDeviceWidth: +-deviceHeight: +-minDeviceHeight: +-maxDeviceHeight: +-orientation: accepting ==CssMediaQueryConstants portrait== or ==CssMediaQueryConstants landscape== + +Accepting fractions as aspect ratios +-aspectRatio: +-minAspectRatio: +-maxAspectRatio: +-deviceAspectRatio: +-minDeviceAspectRatio: +-maxDeviceAspectRatio: + +Accepting integers +-color: the argument describes the number of bits per color component of the output device +-minColor: +-maxColor: +-colorIndex: the argument describes the number of entries in the color lookup table of the output device +-minColorIndex: +-maxColorIndex: +-monochrome: the argument describes the number of bits per pixel in a monochrome frame buffer +-minMonochrome: +-maxMonochrome: +-grid: the argument must be 1 or 0 + +Accepting as argument a ==CssResolution== +-resolution: +-minResolution: +-maxResolution: +-scan: accepting ==CssMediaQueryConstants progressive== or ==CssMediaQueryConstants interlace== + +A new basic type is added: ==CssResolution==. This kind of measures can be created sending the messages ==dpi== (dots per inch) or ==dpcm== (dots per centimeter) to an integer or float. + +Let's see a final example to better understand the media features support: + +[[[ +CascadingStyleSheetBuilder new + declare: [ :cssBuilder | + cssBuilder + declareRuleSetFor: [ :selector | selector id: #oop ] + with: [ :style | style color: CssSVGColors red ] + ] + forMediaMatching: [ :queryBuilder | + queryBuilder + orientation: CssMediaQueryConstants landscape; + resolution: 300 dpi + ]; + build + +]]] + + +produces: + +[[[ +@media all and (orientation: landscape) and (resolution: 300dpi) +{ + #oop + { + color: red; + } +} +]]] + +!! Conclusion + +This concludes this tutorial showing how you can programmatically emit css. + + diff --git a/Voyage/Voyage.pier b/Voyage/Voyage.pier new file mode 100644 index 0000000..9cab098 --- /dev/null +++ b/Voyage/Voyage.pier @@ -0,0 +1,541 @@ +!Persisting Objects with Voyage + + +Voyage is a small persistence framework developed by Esteban Lorenzano, constructed as a small layer between the objects and a persistency mechanism. It is purely object-oriented and has as a goal to present a minimal API to most common development usages. Voyage is a common layer for different backends but currently it supports just two: an ''in-memory'' layer and a backend for the MongoDB database (*http://mongodb.org>http://mongodb.org/*). + +The in-memory layer is useful to prototype applications quickly and for initial development without a database back-end, for example using the Smalltalk image as the persistency mechanism. + +The MongoDB database backend stores the objects into MongoDB: a document-oriented database. In MongoDB each stored entity is a JSON-style document. This document-centric nature allows for persisting complex object models in a fairly straightforward fashion. MongoDB is not an object database, like Gemstone, Magma or Omnibase, so there still is a small gap to be bridged between objects and documents. To achieve this, Voyage contains an Object-Document mapper, i.e. the document database equivalent to an Object-Relational Mapper or ORM. While this mapper does not solve all the known impedance mismatch issues when going from objects to a database, we find that using a document database fits better with the object world than a combination of a ORM and a relational database. This is because document databases tend to provide better support for the dynamic nature of the object world. + +Voyage provides a default way in which objects are stored in the database. Fine-grained configuration of this can be performed using Magritte descriptions. Voyage also includes a query API, which allows specific objects to be retrieved from a MongoDB database. We will discuss each of these features in this text. + +(This text started as a number of blog posts by Esteban Lorenzano, which have been extensively reworked by Johan Fabry, and including additional information shared by Sabine Knoefel and Norbert Hartl. For corrections, comments and questions about this text please send a mail to Johan Fabry. His e-mail address is dcc.uchile.cl@jfabry, but the other way around. ) + +!! Setup + +!!! Load Voyage + +To install Voyage, including support for the MongoDB database, go to the Configurations Browser (in the World Menu/Tools) and load ConfigurationOfVoyageMongo. Or alternatively execute in a workspace: + +[[[ +Gofer it + url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main'; + configurationOf: 'VoyageMongo'; + loadStable. +]]] + +This will load all that is needed to persist objects into a Mongo database. + +!!! Install MongoDB + +Next is to install the MongoDB database. How to do this depends on the operating system, and is outside of the scope of this text. We refer to the *MongoDB website>http://www.mongodb.org/downloads* for more information. + +!!! Create a repository + +In Voyage, all persistent objects are stored in a repository. The kind of repository that is used determines the storage backend for the objects. + +To use the in-memory layer for Voyage, an instance of ==VOMemoryRepository== needs to be created, as follows: + +[[[ +repository := VOMemoryRepository new +]]] + +In this text, we shall however use the MongoDB backend. To start a new MongoDB repository or connect to an existing repository create an instance of ==VOMongoRepository==, giving as parameters the hostname and database name. For example, to connect to the database ==databaseName== on the host ==mongo.db.url== execute the following code: + +[[[ +repository := VOMongoRepository + host: 'mongo.db.url' + database: 'databaseName'. +]]] + +Alternatively, the port to connect to is specified by using the message ==host:port:database:==. Lastly, if authentication is required, the message ==host:database:username:password:== or ==host:port:database:username:password:== needs to be used. + +!!! Singleton Mode and Instance Mode + +Voyage can work in two different modes: + +- Singleton mode: There is an unique repository in the image, which works as a singleton keeping all the data. When you use this mode, you can program using a "behavioral complete" approach where instances respond to a certain vocabulary (see below for more details about vocabulary and usage). +- Instance mode: You can have an undetermined number of repositories living in the image. Of course, this mode requires you to make explicit which repositories you are going to use. + + +By default, Voyage works in instance mode: the returned instance has to be passed as an argument to all database API operations. Instead of having to keep this instance around, a convenient alternative is to use Singleton mode. Singleton mode removes the need to pass the repository as an argument to all database operations. To use Singleton mode, execute: + +[[[ +repository enableSingleton. +]]] + +@@note Pay attention: Only one repository can be the singleton, hence executing this line will remove any other existing repositories from Singleton mode! + +@@note In this document, we are going to cover Voyage in Singleton mode, but using it in Instance mode is straightforward as well. See the ==persistence== protocol of ==VORepository== for more information. + +!!! Voyage API + +The following two tables show a representative subset of the API of Voyage. These methods are defined on ==Object== and ==Class==, but will only truly perform work if (instances of) the receiver of the message is a Voyage root. See the ==voyage-model-core-extensions== persistence protocol on both classes for the full API of Voyage. + +First we show Singleton mode: +||save || stores an object into repository (insert or update) +||remove || removes an object from repository +||removeAll ||removes all objects of class from repository +||selectAll ||retrieves all objects of some kind +||selectOne: ||retrieves first object that matches the argument +||selectMany: || retrieves all objects that matches the argument + + +Second is Instance mode. In Instance mode, the first argument is always the repository on which to perform the operation. +||save:|| stores an object into repository (insert or update) +||remove: ||removes an object from repository +||removeAll: ||removes all objects of class from repository +||selectAll: ||retrieves all objects of some kind +||selectOne:where: ||retrieves first object that matches the where clause +||selectMany:where: || retrieves all objects that matches the where clause + +!!! Resetting or dropping the database connection + +In a deployed application, there should be no need to close or reset the connection to the database. Also, Voyage re-establishes the connection when the image is closed and later reopened. + +However, when developing, resetting the connection to the database may be needed to reflect changes. This is foremost required when changing storage options of the database (see section *enhancing*). Performing a reset is achieved as follows: + +[[[ +VORepository current reset. +]]] + +In case the connection to the database needs to be dropped, this is performed as follows: + +[[[ +VORepository setRepository: nil. +]]] + +!!Storing objects + +To store objects, the class of the object needs to be declared as being a ''root of the repository''. All repository roots are points of entry to the database. Voyage stores more than just objects that contain literals. Complete trees of objects can be stored with Voyage as well, and this is done transparently. In other words, there is no need for a special treatment to store trees of objects. However, when a graph of objects is stored, care must be taken to break loops. In this section we discuss such basic storage of objects, and in section *enhancing* on Enhancing Storage we show how to enhance and/or modify the way objects are persisted. + +!!!Basic storage + +Let's say we want to store an Association (i.e. a pair of objects). To do this, we need to declare that the class ==Association== is storable as a root of our repository. To express this we define the class method ==isVoyageRoot== to return true. + +[[[ +Association class>>isVoyageRoot + ^true +]]] + +We also define the name of the collection that will be used to store documents with the ==voyageCollectionName== class method. + +[[[ +Association class>>voyageCollectionName + ^ 'Associations' +]]] + +Then, to save an association, we need to just send it the ==save== message: + +[[[ +anAssociation := #answer->42. +anAssociation save. +]]] + +This will generate a collection in the database containing a document of the following structure. + +[[[ +{ + "_id" : ObjectId("a05feb630000000000000000"), + "#instanceOf" : "Association", + "#version" : NumberLong("3515916499"), + "key" : 'answer', + "value" : 42 +} +]]] + +The collection in the database will be named ''association'', for the name of the root class. By default, Voyage creates a MongoDB collection for each root class. + +The stored data keeps some ''extra information'' to allow the object to be correctly reconstructed when loading: + +- ==instanceOf== records the Class of the stored instance. The collection can also contain subclasses of the class that was declared as a Voyage root, so to properly reconstruct instances the reference to the class is needed. +- ==version== keeps a marker of the version of the object that is committed. This property is used internally by Voyage for refreshing cached data in the application. Without a ==version== field the application would always refresh the object with the data from the database, losing a lot of performance). + +Note that the documents generated by Voyage are not directly visible using Voyage itself, as the goal of Voyage is to abstract away from the document structure. To see the actual documents you need to access the database directly. For MongoDB this can be done through the ==mongo== command line interface, or a GUI tool such as RoboMongo (Multi-Platform) MongoHub (for Mac), or RockMongo (in php). + +!!!Embedding objects +In general, objects are not so simple as associations of literals, but more complex, containing other complex object instances inside, leading to a tree of objects. We show now how these can be stored. + +For example, let's say that we want to store rectangles and that each rectangle contains two points. To achieve this, we specify that the ==Rectangle== class is a document root: + +[[[ +Rectangle class>>isVoyageRoot + ^true +]]] + +This allows rectangles to be saved to the database, for example as shown by this snippet: + +[[[ +aRectangle := 42@1 corner: 10@20. +aRectangle save. +]]] + +This will add a document to the database with this structure: + +[[[ +{ + "_id" : ObjectId("ef72b5810000000000000000"), + "#instanceOf" : "Rectangle", + "#version" : NumberLong("2460645040"), + "origin" : { + "#instanceOf" : "Point", + "x" : 42, + "y" : 1 + }, + "corner" : { + "#instanceOf" : "Point", + "x" : 10, + "y" : 20 + } +} +]]] + + + +!!!Referencing other roots +@referencing + +Sometimes the objects are trees that contain other root objects. For instance, you could want to keep users and roles as roots, i.e. in different collections, and a user has a collection of roles. If the embedded objects (the roles) are root objects, Voyage will store references to these objects instead of including them in the document. + +Returning to our rectangle example, let's suppose we want to keep the points in a separate collection. In other words, now the points will be referenced instead of embedded. + +After we add ==isVoyageRoot== to ==Point class==, and save the rectangle, in the collection ''rectangle'' we get the following document: + +[[[ +{ + "_id" : ObjectId("7c5e772b0000000000000000"), + "#instanceOf" : "Rectangle", + "#version" : 423858205, + "origin" : { + "#collection" : "point", + "#instanceOf" : "Point", + "__id" : ObjectId("7804c56c0000000000000000") + }, + "corner" : { + "#collection" : "point", + "#instanceOf" : "Point", + "__id" : ObjectId("2a731f310000000000000000") + } +} +]]] + +In addition to this, in the collection ''point'' we also get the two following entities: + +[[[ +{ + "_id" : ObjectId("7804c56c0000000000000000"), + "#version" : NumberLong("4212049275"), + "#instanceOf" : "Point", + "x" : 42, + "y" : 1 +} + +{ + "_id" : ObjectId("2a731f310000000000000000"), + "#version" : 821387165, + "#instanceOf" : "Point", + "x" : 10, + "y" : 20 +} +]]] + +!!! Breaking Cycles in Graphs + +When the objects to be stored contain a graph of embedded objects instead of a tree, i.e. when there are cycles in the references that the embedded objects have between them, cycles between these embedded objects must be broken. If not, storing the objects will cause an infinite loop. The most straightforward solution is to declare one of the objects causing the cycle as a Voyage root. This effectively breaks the cycle at storage time, avoiding the infinite loop. + +For example, in the rectangle example say we have a label inside the rectangle, and this label contains a piece of text. The text also keeps a reference to the label in which it is contained. In other words there is a cycle of references between the label and the text. This cycle must be broken in order to be able to persist the rectangle. To do this, either the label or the text must be declared as a Voyage root. + +An alternative solution to break cycles, avoiding the declaration of new voyage roots, is to declare some fields of objects as transient and define how the graph must be reconstructed at load time. This will be discussed in the following section. + +!! Enhancing Storage +@enhancing + +How objects are stored can be changed by adding Magritte descriptions to their classes. In this section, first we will talk about configuration options for the storage format of the objects, and then we will treat more advanced loading and saving of attributes, which can be used, e.g. to break cycles in embedded objects. + +!!!Configuring Storage + +Consider that, continuing with the rectangle example but using embedded points, we add the following storage requirements: + +- We need to use a different collection named ''rectanglesForTest'' instead of ''rectangle''. +- We only store instances of the ==Rectangle== class in this collection, and therefore the ==instanceOf== information is redundant. +- The ==origin== and ==corner== attributes are always going to be points, so the ==instanceOf== information there is redundant as well. + +We use Magritte descriptions with specific pragmas to declare properties of the container of the Mongo descriptions of that class (all Magritte descriptions are kept in containers) and to describe both the ==origin== and ==corner== attributes. + +The method ==mongoContainer== is defined as follows: First it uses the ==== pragma to state that it describes the container to be used for this class. Second it returns a specific ==VOMongoContainer== instance. This instance is configured such that it uses the ''retanglesForTest'' collection in the database, and that it will only store ==Rectangle== instances. Note that it is not required to specify both configuration lines. It is equally valid to only declare that the collection to be used is ''retanglesForTest'', or only specify that the collection contains just ==Rectangle== instances. + +[[[ +Rectangle class>>mongoContainer + + + ^VOMongoContainer new + collectionName: 'rectanglesForTest'; + kind: Rectangle; + yourself +]]] + +The two other methods use the pragma ==== and return a Mongo description that is configured with their respective attribute name and kind, as follows: + +[[[ +Rectangle class>>mongoOrigin + + + ^VOMongoToOneDescription new + attributeName: 'origin'; + kind: Point; + yourself +]]] + +[[[ +Rectangle class>>mongoCorner + + + ^VOMongoToOneDescription new + attributeName: 'corner'; + kind: Point; + yourself +]]] + +A saved rectangle, now in the ''rectanglesForTest'' collection, will look more or less as follows: + +[[[ +{ + "_id" : ObjectId("ef72b5810000000000000000"), + "#version" : NumberLong("2460645040"), + "origin" : { + "x" : 42, + "y" : 1 + }, + "corner" : { + "x" : 10, + "y" : 20 + } +} +]]] + +Other configuration options for attribute descriptions are: + +- ==beEager== declares that the referenced instance is to be loaded eagerly (the default is lazy). +- ==beLazy== declares that referenced instances are loaded lazily. +- ==convertNullTo:== when retrieving an object whose value is Null (==nil==), instead return the result of evaluating the block passed as argument. + +For attributes which are collections, the ==VOMongoToManyDescription== needs to be returned instead of the ==VOMongoToOneDescription==. All the above configuration options remain valid, and the ==kind:== configuration option is used to specify the kind of values the collection contains. ==VOMongoToManyDescription== provides a number of extra configuration options: + +- ==kindCollection:== specifies the class of the collection that is contained in the attribute. +- ==convertNullToEmpty== when retrieving a collection whose value is Null (==nil==), it returns an empty collection. + +!!!Custom Loading and Saving of Attributes + +It is possible to write specific logic for transforming attributes of an object when written to the database, as well as when read from the database. This can be used, e.g., to break cycles in the object graph without needing to declare extra Voyage roots. To declare such custom logic, a ==MAPluggableAccessor== needs to be defined that contains Smalltalk blocks for reading the attribute from the object and writing it to the object. Note that the names of these accessors can be counter-intuitive: the ==read:== accessor defines the value that will be ''stored'' in the database, and the ==write:== accessor defines the transformation of this ''retrieved'' value to what is placed in the object. This is because the accessors are used by the Object-Document mapper when reading the object to store it to the database and when writing the object to memory, based on the values obtained from the database. + +Defining accessors allows, for example, a ==Currency== object that is contained in an ==Amount== to be written to the database as its' three letter abbreviation (EUR, USD, CLP, ...). When loading this representation, it needs to be converted back into a Currency object, e.g. by instantiating a new Currency object. This is achieved as follows: + +[[[ +Amount class>>#mongoCurrency + + + ^ VOMongoToOneDescription new + attributeName: 'currency'; + accessor: (MAPluggableAccessor + read: [ :amt | amt currency abbreviation ] + write: [ :amt :value | amt currency: (Currency fromAbbreviation: value)]); + yourself +]]] + +Also, a post-load action can be defined for an attribute or for the containing object, by adding a ==postLoad:== action to the attribute descriptor or the container descriptor. This action is a one-parameter block, and will be executed after the object has been loaded into memory with as argument the object that was loaded. + +Lastly, attributes can be excluded from storage (and hence retrieval) by returning a ==VOMongoTransientDescription== instance as the attribute descriptor. This allows to place cut-off points in the graph of objects that is being saved, i.e. when an object contains a reference to data that should not be persisted in the database. This may also be used to break cycles in the stored object graph. It however entails that when retrieving the graph from the database, attributes that contain these objects will be set to ==nil==. To address this, a post-load action can be specified for the attribute descriptor or the container descriptor, to set these attributes to the correct values. + + +!! Querying in Voyage + +Voyage allows to selectively retrieve object instances though queries on the database. When using the in-memory layer, queries are standard Smalltalk blocks. When using the MongoDB back-end, the MongoDB query language is used to perform the searches. To specify these queries, MongoDB uses JSON structures, and when using Voyage there are two ways in which these can be constructed. MongoDB queries can be written either as blocks or as dictionaries, depending on their complexity. In this section, we first discuss both ways in which queries can be created, and we end the section by talking about how to execute these queries. + +!!! Basic object retrieval using blocks or MongoQueries + +The most straightforward way to query the database is by using blocks when using the in-memory layer or MongoQueries when using the MongoDB back-end. In this discussion we will focus on the use of MongoQueries, as the use of blocks is standard Smalltalk. + +MongoQueries is not part of Voyage itself but part of the MongoTalk layer that Voyage uses to talk to MongoDB. MongoTalk was made by Nicolas Petton and provides all the low-level operations for accessing MongoDB. MongoQueries transforms, within certain restrictions, regular Pharo blocks into JSON queries that comply to the form that is expected by the database. In essence, MongoQueries is an embedded Domain Specific Language to create MongoDB queries. Using MongoQueries, a query looks like a normal Pharo expression (but the language is much more restricted than plain Smalltalk). + +Using MongoQueries, the following operators may be used in a query: + +|Operand|Usage| +|==< <= > >= = ~= ==|Regular comparison operators| +|==&==|AND operator| +| ==\|== |OR operator| +|==not==|NOT operator| +|==at:==|Access an embedded document| +|==where:==|Execute a Javascript query| + +For example, a query that selects all elements in the database whose name is ==John== is the following: + +[[[ +[ :each | each name = 'John'] +]]] + + + +A slightly more complicated query is to find all elements in the database whose name is ==John== and the value in ==orders== is greater than 10. + +[[[ +[ :each | (each name = 'John') & (each orders > 10 ) ] +]]] + + +Note that this way of querying only works for querying values of the object but not values of references to other objects. +For such case you should build your query using id as traditionally done in relational database. But the best solution to be in the Mongo spirit, is in fact to revisit the object model to avoid relationships expressed with foreign keys. + + +!!!!Using the ==at:== message to access embedded documents + +Since MongoDB stores documents of any complexity, it is common that one document is composed of several embedded documents, for example: + +[[[ +{ + "origin" : { + "x" : 42, + "y" : 1 + }, + "corner" : { + "x" : 10, + "y" : 20 + } +} +]]] + +In this case, to search for objects by one of the embedded document elements, the message ""==at:=="", and the field separator """.""" needs to be used. For example, to select all the rectangles whose origin x value is equal to 42, the query is as as follows. + +[[[ +[ :each | (each at: 'origin.x') = 42 ] +]]] + +!!!!Quering with elements from another root document + +With no-SQL databases, it is not possible to query on multiple collections (equivalent of JOIN in SQL). You have two options: alter your schema to use embedded documents or write application-level code to reproduce the JOIN behavior. The later option can be done with two queries: + +[[[ +color := Color selectOne: [ :each | each name = 'red' ] +Point selectOne: [ :each | (each at: 'color.__id') = color voyageId ] +]]] + + +!!!!Using ==where:== message to perform Javascript comparisons + +To perform queries which are outside the capabilities of MongoQueries or even the MongoDB query language, MongoDB provides a way to write queries directly in Javascript using the ==$where== operand. This is also possible in MongoQueries by sending the ==where:== message: + +In the following example we repeat the previous query but a Javascript expression instead. The expression is =='this.origin.x \== 42'== and to use it, the query is as follows: + +[[[ +[ :each | each where: 'this.origin.x == 42' ]. +]]] + +More complete documentation about the use of ==$where== is in the *MongoDB where documentation>http://docs.mongodb.org/manual/reference/operator/where/#op._S_where*. + + +!!!Using JSON queries + +If MongoQueries does not allow a query to be expressed, a JSON query can be created directly. JSON queries are the MongDB query internal representation, and can be created straightforwardly in Voyage. In a nutshell: a JSON structure is mapped to a dictionary with pairs. In these pairs the key is a string and the value can be a primitive value, a collection or another JSON structure (i.e. another dictionary). To create a query, we simply need to create a dictionary that satisfies these requirements. Note that the use of JSON queries is strictly for when using the MongoDB back-end. Other back-ends, \eg~the in-memory layer, do not provide support for the use of JSON queries. + +For example, the first example of the use of MongoQueries is written as a dictionary as follows: + +[[[ +{ 'name' -> 'John' } asDictionary +]]] + +Dictionary pairs are composed with AND semantics. The following example repeats the second example of the use of MongoQueries; selecting the elements having ==John== as name AND whose ==orders== value is greater than 10. + +[[[ +{ + 'name' -> 'John'. + 'orders' -> { '$gt' : 10 } asDictionary +} asDictionary +]]] + +To construct the "greater than" statement, a new dictionary needs to be created that uses the MongoDB ==$gt== query selector to express the greater than relation. For the list of available query selectors we refer to the *MongoDB Query Selectors documentation>http://docs.mongodb.org/manual/reference/operator/query/#query-selectors*. + + +!!!!Using dot notation to access embedded documents + +To access values embedded documents with JSON queries, the dot notation is used. For example, a reformulation of the MongoQueries embedded documents example is the following: + +[[[ +{ + 'origin.x' -> {'$eq' : 42} asDictionary +} asDictionary +]]] + + +!!!!Expressing OR conditions in the query + + +To express an OR condition, a dictionary whose key is =='$or'== and whose values are the expression of the condition is needed. The following example shows how to select all objects whose name is ==John== that have more than ten orders OR objects whose name is not ==John== and has ten or less orders: + +[[[ +{ '$or' : + { + { + 'name -> 'John'. + 'orders' -> { '$gt': 10 } asDictionary + } asDictionary. + { + 'name -> { '$ne': 'John'} asDictionary. + 'orders' -> { '$lte': 10 } asDictionary + } asDictionary. + }. +} asDictionary. +]]] + + +!!!!Going beyond MongoQueries features + +Using JSON queries allows to use features that are not present in MongoQueries, for example the use of regular expression. Below is a query that specifies all documents with a ==fullname.lastName== that starts with a letter "D" and then matches all the attributes of the ==fullname== embedded document, by using the Mongo expressions ==$regexp== and ==$options$== as follows: + +[[[ +{ + 'fullname.lastName' -> { + '$regexp': '^D.*'. + '$options': 'i'. + } asDictionary. +} asDictionary. +]]] + +This example only briefly illustrates the power of JSON queries. Many more different queries can be constructed, and the complete list of operators and usages is in the *MongoDB operator documentation>http://docs.mongodb.org/manual/reference/operator* + + +!!!Executing a query + +Voyage has a group of methods to perform search that can be more or less mapped to regular collection methods. For the examples, we will use stored Point instances we have seen before (in the section *referencing* other roots) to illustrate the use of these methods. Note that all queries in this section can be either MongoQueries or JSON queries, unless otherwise specified. + +!!!!Basic object retrieval + +The following methods provide basic object retrieval. + +- ""==selectAll=="" Retrieves all documents in the corresponding database collection. For example, ==Point selectAll== will return all Points. +- ""==selectOne:=="" Retrieves one document that matches the query. This maps to a ==detect:== method and takes as argument a query specification (either a MongoQuery or a JSON Query). For example, ==Point selectOne: [:each | each x = 42] == or ==Point selectOne: { 'x' -> 42 } asDictionary== +- ""==selectMany:=="" Retrieves all the documents that match the query. This maps to a ==select:== method and takes as argument a query specification, like above. + +!!!!Limiting object retrieval and sorting + +The methods that query the database look similar to their equivalent in the Collection hierarchy. However unlike regular collections which can operate fully on memory, often Voyage collection queries need to be customized in order to optimize memory consumption and/or access speed. This is because there can be literally millions of documents in each collection, surpassing the memory limit of Smalltalk, and also the database searches have a much higher performance than the equivalent code in Smalltalk. + +The first refinement to the queries consist in limiting the amount of results that are returned. Of the collection of all the documents that match, a subset is returned that starts at the index that is given as argument. This can be used to only retrieve the first N matches to a query, or go over the query results in smaller blocks to avoid out of memory issues in the Smalltalk image. + + +- ""==selectMany:limit:=="" Retrieves a collection of objects from the database that match the query, up to the given limit. For example, ==Point selectMany: [:each | each x = 42] limit: 10 == +- ""==selectMany:limit:offset:=="" Retrieves a collection of objects from the database that match the query. The first object retrieved will be at the offset position plus one of the results of the query, and up to limit objects will be returned. For example, if the above example matched 25 points, the last 15 points will be returned by the query ==Point selectMany: [:each | each x = 42] limit: 20 offset: 10== (any limit argument greater than 15 will do for this example). + +The second customization that can be performed is to sort the results. To use this, the class ==VOOrder== provides constants to specify ==ascending== or ==descending== sort order. + +- ""==selectAllSortBy:=="" Retrieves all documents, sorted by the specification in the argument, which needs to be a JSON query. For example, == Point selectAllSortBy: { #x -> VOOrder ascending} asDictionary == returns the points in ascending x order. +- ""==selectMany:sortBy:=="" Retrieves all the documents that match the query and sorts them. For example, == Point selectMany: { 'x' -> 42 } SortBy: { #y -> VOOrder descending} asDictionary == returns the points where x is 42, in descending y order. +- ""==selectMany:sortBy:limit:offset:=="" Provides for specifying a limit and offset to the above query. + + +!! Conclusion + +In this chapter we presented Voyage, a persistence framework whose strength is the presence of a basic object-document mapper and a back-end to the MongoDB database. We have shown how to store objects in, and remove object from the database, and how to optimise the storage format. This was followed by a discussion of querying the database; showing the two ways in which queries can be constructed and detailing how queries are ran. + + +% Local Variables: +% eval: (flyspell-mode 1) +% End: diff --git a/WebApp/WebApp.pier b/WebApp/WebApp.pier new file mode 100644 index 0000000..dc5cd66 --- /dev/null +++ b/WebApp/WebApp.pier @@ -0,0 +1,887 @@ +!Building and deploying your first web app with Pharo + +-First version: July 10, 2012 by Sven Van Caekenberghe +-Second version: September 10, 2012 by Stéphane Ducasse + +There are lots of ways to get something on the Web today. However, it remains important that you understand the actual mechanics of building and deploying a web application. This guide explains how to build and deploy your first web application using Pharo. +Of course, there are an infinite number of ways to make a web app. Even in Pharo, there are multiple frameworks approaching this problem, most notably Seaside, AIDAweb and Iliad. Here, we'll be using the foundational framework called Zinc HTTP Components. By doing so, we'll be touching the fundamentals of HTTP and web apps. + +This chapter could have been called ''understanding HTTP fundamentals through Zinc HTTP Components''. + +Using nice objects, abstracting each concept in HTTP and related open standards, the actual code will be easier than you might expect. +The dynamic, interactive nature of Pharo combined with its rich IDE and library will allow us to do things that are nearly impossible using other technology stacks. By chronologically following the development process, you will see the app growing from something trivial to the final result. Finally, we will save our source code in a repository and deploy for real in the cloud. + +Let's get started. + + + +!! The Web App + +The web application that we are going to build will show a picture and allow users to change the picture by uploading a new one as shown by Figure *imageweb*. Because we want to focus on the basic mechanics, the fundamentals as well as the build and deploy process, there are some simplifications. There will be one picture for all users, no login and we will store the picture in memory. + ++A simple web application>file://figures/image-web-app.png|width=80|label=imageweb+ + +In our implementation, ==/image== will serve an HTML page containing the image and a form. To serve the raw image itself, we'll add a parameter, like ==/image?raw=true==. These will be GET HTTP requests. The form will submit its data to ==/image== as a POST request. + + +!! Downloading Pharo + +Go to http://www.pharo.org and download the whole self-contained package for your platform, it is just 12 to 14 MB. Select the released version 2.0. Although not recommended for beginners, current development version 3.0 will do just fine as well. Double-click and you enter the Pharo world. + ++Pharo>file://figures/pharo-in-action.png|width=80|label=pharo-in-action+ + + +Pharo is an incredible sophisticated and deep environment and toolset. The Pharo by Example book available at: http://www.pharobyexample.org is probably the best way to get started if all this is totally new to you. In what follows we assume you at least read the first chapter, 'A Quick Tour of Pharo'. + +!! Running an HTTP server + +Open a Workspace, type and execute +[[[ +ZnServer startDefaultOn: 8080. +]]] + +Now open the address ==http://localhost:8080== in your favorite browser. You should get the default welcome page of Zinc HTTP Components. If you visit ==http://localhost:8080/help== you will see a list of all available pages. Now add the following line to your workspace and execute it: + +[[[ +ZnServer default logToTranscript. +]]] + +Next open the Transcript and visit or reload a page. You should see log output like this + +[[[ +2013>07>07 00:22:49 479147 D Executing request/response loop +2013>07>07 00:22:49 479147 I Read a ZnRequest(GET /) +2013>07>07 00:22:49 479147 T GET / 200 977B 2ms +2013>07>07 00:22:49 479147 I Wrote a ZnResponse(200 OK text/html;charset=utf>8 977B) +]]] + +You can see the server entering the request/response loop for a certain connection/thread. A request is read and a response is written. Let's have a look under the hood. Put the server in debug mode and inspect it like this + +[[[ +ZnServer default debugMode: true; inspect. +]]] + +Visit and reload a page. Now you can use the inspector to explore the actual lastRequest and lastResponse objects. Pretty cool, right? + +Tomplete our little tour, let's try one more thing. We can execute any request programmatically as well, using an HTTP client. To visit a page, try inspecting the result of the following expression: + +[[[ +ZnClient new get:'http://localhost:8080/random' +]]] + +If you would look inside the client object, you would find similar request and response objects - this makes total sense since the client talks to the server and vice versa, over the network. If you want, you can stop the server executing the following expression: + +[[[ +ZnServer stopDefault. +]]] + +If you are curious, please consult the Zinc HTTP Components documentation available at: @@ http @@ + + + +!! Saying hello world + +Let's lay the groundwork for our new web application by making a version that only says 'Hello World!'. We'll be extending the web app gradually until we reach our functional goal. + +Open the Nautilus System Browser and create a new package (right click in the first column) called something like "'MyFirstWebApp'". Now create a new class (right click in the second column) with the same name, "MyFirstWebApp". You will be given a template: edit 'NameOfSubclass' and accept by clicking 'OK'. Your definition should now appear in the bottom pane. + +[[[ +Object subclass: #MyFirstWebApp + instanceVariableNames: '' + classVariableNames: '' + poolDictionaries: '' + category: 'MyFirstWebApp' +]]] + +Any object can be a web app, it only has to respond to a message called ==handleRequest:== to answer a response based on a request. Now add the following method: + +[[[ +MyFirstWebApp>>handleRequest: request + request uri path = #image + ifFalse: [ ^ ZnResponse notFound: request uri ]. + ^ ZnResponse ok: (ZnEntity text: 'Hello World!') +]]] + +Create a new protocol called 'public' (by right-clicking in the third column). When the new protocol is selected, a new method template will appear in the bottom pane. Overwrite the whole template with the code above and accept it (as shown Figure *pharo-in-action*). + ++Defining a first version of the application.>file://figures/pharo-in-action.png|width=80|label=pharo-in-action+ + +What we do here is to look at the incoming request to make sure the URI path is ==/image== which will be the final name of our web app. If not, we return a Not Found (code 404) response. If so, we create and return an OK response (code 200) with a simple text entity as body or payload. + +[[[ +MyFirstWebApp>>value: request + ^ self handleRequest: request +]]] + +Now we define the methos ==value:== to make it an alias of ==handleRequest:== - this is needed so our web app object can be used more flexibly. To test our web app, we'll add it as one of the pages of the default server, like this + +[[[ +ZnServer startDefaultOn: 8080. +ZnServer default delegate map: #image to: MyFirstWebApp new. +]]] + +The second expression adds a route from ==/image== to an instance of our web app object. If all is well, ==http://localhost:8080/image== should show your friendly message. Note how we are not even serving HTML, just plain text. + +Try changing the text. Try putting a breakpoint in ==MyFirstWebApp>>handleRequest:== (right-click on the method name in the fourth column) and inspecting things. Then just continue the execution. Note how this is a live environment: you make a little change and it is immediately used, you can look into the actual request and response objects moving around (as shown in Figures *debugger1* and *debugger2*. + + ++Using the debugger to navigate execution>file://figures/breakpoint-1.png|width=80|label=debugger1+ ++Using the debugger to navigate execution>file://figures/breakpoint-2.png|width=80|label=debugger2+ + + +Leave the server running. If you want you can enable logging again, or switch to debug mode and inspect the server instance. Don't forget to remove any breakpoints you set. + +!! Serving an HTML page + +HTML generation and/or using templates can be done with some of the higher level frameworks, here we'll manually compose our HTML. Go ahead and add a new method, #html, while changing a previous one slightly + +MyFirstWebApp>>html + ^ 'Image + +

Image

+ ' + +MyFirstWebApp>>handleRequest: request + request uri path = #image + ifFalse: [ ^ ZnResponse notFound: request uri ]. + ^ ZnResponse ok: (ZnEntity html: self html) + +Accept the above two methods and test ==http://localhost:8080/image== again to make sure you now see a real HTML page. + +You have probably noted the red exclamation mark icon in front of our class name in the browser. This is an indication that we have no class comment, which is not good: documentation is important. Click the 'Comment' button and write some documentation. You can also use the class comment as a notepad for yourself, saving useful expressions that you can later execute in place. + +!! Serving an Image + +Images for the purpose of our web app can be any of three types: GIF, JPEG or PNG. We will store them in memory as an entity, an object wrapping the actual bytes together with a mime type. + +To simplify our app, we will arrange things so that we always start with a default image, then we always have something to show. Let's add a little helper, ==downloadPharoLogo== + + + +[[[ +MyFirstWebApp>>downloadPharoLogo + ^ ZnClient new + beOneShot; + get: 'http://www.pharo-project.org/images/pharo.png'; + entity +]]] + + +Quickly test the code by selecting the method body (not including the name) and inspecting the result. You should get an image entity back. Now add the accessor ==image== defined as follow: + + +[[[ +MyFirstWebApp>>image + ^ image ifNil: [ image := self downloadPharoLogo ] +]]] + + +When you try to accept this method, you will get an error. We are using an unknown variable, image. Select the option to automatically declare a new instance variable and we are good. + +Remember that we decided we were going to serve the raw image itself using a query variable, like ==/image?raw=true==. +Make the following modification to existing methods and add a new one + +[[[ +MyFirstWebApp>>html + ^ 'Image + +

Image

+ + ' + +MyFirstWebApp>>handleRequest: request + request uri path = #image + ifFalse: [ ^ ZnResponse notFound: request uri ]. + ^ self handleGetRequest: request + +MyFirstWebApp>>handleGetRequest: request + ^ (request uri queryAt: #raw ifAbsent: [ nil ]) + ifNil: [ ZnResponse ok: (ZnEntity html: self html) ] + ifNotNil: [ ZnResponse ok: self image ] +]]] + +We extended our HTML with a IMG element. We delegate some of our request handling to a new method, ==handleGetRequest:== where we inspect +the incoming URI. If it has a non empty query variable raw we serve the raw image directly, else we serve the HTML page like before. + +Check it out: you should now see an image in the browser when visiting ==http://localhost:8080/image==! + +!! Uploading a new image + + +Interaction is what differentiates a web site from a web application. We will now add the ability for users to upload a new image to change the one on the server. To add this ability we need to use an HTML form. Let's change our HTML one final time. + + +[[[ +MyFirstWebApp>>html + ^ 'Image + +

Image

+ +
+
+

Change the image:

+ + +
+ ' +]]] + + +The user will be able to select a file on the local disk for upload. When s/he click the Upload submit button, the web browser will send an HTTP POST to the action URL, ==/image==, encoding the form contents using a technique called multi-part form-data. With the above change, you will be able to see the form, its just won't work, yet. + +In our request handling, we have to distinguish between GET and POST requests. Change ==handleRequest:== to its final form. + + +[[[ +MyFirstWebApp>>handleRequest: request + request uri path = #image + ifTrue: [ + request method = #GET + ifTrue: [ ^ self handleGetRequest: request ]. + request method = #POST + ifTrue: [ ^ self handlePostRequest: request ] ]. + ^ ZnResponse notFound: request uri +]]] + + + +Now we have to add an implementation of ==handlePostRequest:== to accept the uploaded image and change the current one. + +[[[ +MyFirstWebApp>>handlePostRequest: request + | part newImage | + part := request entity partNamed: #file. + newImage := part entity. + image := newImage. + ^ ZnResponse redirect: #image +]]] + +We start with a simple version without error handling. The entity of the incoming request is a multi-part form-data object containing named parts. Each part, such as the file part, contains another sub-entity. In our case, the uploaded image. Note also how the response to the POST is a redirect to our main page. You should now have a fully functional web app. Go and try it out! + +We have taken a bit of a shortcut in the code above. It is pretty dangerous to just accept what is coming in from the internet without doing some checking. Here is a version that does that. + +[[[ +MyFirstWebApp>>handlePostRequest: request + | part newImage badRequest | + badRequest := [ ^ ZnResponse badRequest: request ]. + (request hasEntity + and: [ request contentType matches: ZnMimeType multiPartFormData ]) + ifFalse: badRequest. + part := request entity + partNamed: #file + ifNone: badRequest. + newImage := part entity. + (newImage notNil + and: [ newImage contentType matches: 'image/*' asZnMimeType ]) + ifFalse: badRequest. + image := newImage. + ^ ZnResponse redirect: #image +]]] + +Our standard response when something is wrong will be to return a Bad Request (code 400). We define this behaviour to a local variable so that we can reuse it multiple times over. The first test makes sure there actually is an entity in the POST request and that it is of the correct type. Next we handle the case when there is no file part. Finally, we make sure the file part is actually an image (JPEG, PNG or GIF) by matching with the wildcard ==image/*== mime type. + +If you are curious, set a breakpoint in the method and inspect the request object of an actual request. You can learn an awful lot from looking at live objects. + +!! Live debugging +Let's make a deliberate error in our code. Change ==handlePostRequest:== so that the last line reads like + +[[[ +^ ZnResponse redirectTo: #image +]]] + + +The compiler will already complain, ignore the warning and accept the code anyway. Try uploading a new image. The debugger will pop up telling you that ==ZnResponse== does not understand ==redirectTo:== and show you the offending code. You could fix the code and try uploading again to see if it works as shown in Figure *dnu*. + + ++Debugging.>file://figures/dnu.png|width=80|label=dnu+ + + +But we can do better! Just fix the code and accept it. Now you can restart and proceed the execution. The same request is still active and the server will now do the correct thing. Have a look at your web browser: you will see that your initial action, the upload, that first initially hung, has now succeeded. + +Up to now, the suggestion was that you can use the debugger and inspector tools to look at requests and responses. But you can actually change them while they are happening ! Prepare for our experiment by making sure that you change the image to be different from the default one. Now set a breakpoint in ==handleGetRequest:== and reload the main page. There will be two requests coming in: the first one for ==/image== and the second one for ==/image?raw=true==. Proceed the first one. + + ++Live change.>file://figures/live-change.png|width=80|label=live+ + + +Now, with the execution being stopped for the second request, click on the image instance variable in the bottom left pane. The pane next to it will show some image entity. Select the whole contents and replace it with ==self downloadPharoLogo== and accept the change. Now proceed the execution. Your previously uploaded image is gone, replaced again by the default Pharo logo. We just changed an object in the middle of the execution. Imagine doing all your development like that, having a real conversation with your application, while you are developing it. Be warned though: once you get used to this, it will be hard to go back. + +!! Image Magic + +The abilities to look at the requests and responses coming in and going out of the server, to set breakpoints, to debug live request without redoing the user interaction or to modify data structure live are already great and quite unique. But there is more. +Pharo is not just a platform for server applications, it can be used to build regular applications with normal graphics as well. In fact, it is very good at it. That is why it has built-in support to work with JPEG, GIF or PNG. + +Would it not be cool to be able to actually parse the image that we were manipulating as an opaque collection of bytes up till now? To make sure it is real. To look at it while debugging. Turns out this is quite easy. Are you ready for some image magick, pun intended? + +The Pharo object that represents images is called a form. There are objects called ==GIFReadWriter==, ==PNGReadWriter== and ==JPEGReadWriter== that can parse bytes into forms. Add two helper methods, ==formForImageEntity:== and ==form== + +[[[ +MyFirstWebApp>>formForImageEntity: imageEntity + | imageType parserClassName parserClass parser | + imageType := imageEntity contentType sub. + parserClassName := imageType asUppercase, #ReadWriter. + parserClass := Smalltalk globals at: parserClassName asSymbol. + parser := parserClass on: imageEntity readStream. + ^ parser nextImage + +MyFirstWebApp>>form + ^ self formForImageEntity: self image + +]]] + +What we do is use the sub type of the mime type, like "png" in image/png, to find the parser class. Then we instantiate a new parser on a read stream on the actual bytes and invoke the parser with sending ==nextImage==, which will return a form. The ==form== method makes it easy to invoke all this logic on our current image. + +Now we can have a look at, for example, the default image like this +[[[ +MyFirstWebApp new form asMorph openInWindow. +]]] + +Obviously you can do this while debugging too. We can also use the image parsing logic to improve our error checking even further. Here is the final version of ==handlePostRequest:== + + +[[[ +MyFirstWebApp>>handlePostRequest: request + | part newImage badRequest | + badRequest := [ ^ ZnResponse badRequest: request ]. + (request hasEntity + and: [ request contentType matches: ZnMimeType multiPartFormData ]) + ifFalse: badRequest. + part := request entity + partNamed: #file + ifNone: badRequest. + newImage := part entity. + (newImage notNil + and: [ newImage contentType matches: 'image/*' asZnMimeType ]) + ifFalse: badRequest. + [ self formForImageEntity: newImage ] +on: Error + do: badRequest. + image := newImage. + ^ ZnResponse redirect: #image +]]] + +Before making the actual assignment of the new image to our instance variable we added an extra expression. We try parsing the image. We are not interested in the result, but we do want to reply with a bad request when the parsing should fail. + +Once we have a form object, the possibilities are almost endless. You can query a form for the its size, depth and other elements. You can manipulate the form in various ways: scaling, resizing, rotating, flipping, cropping, compositing. And you can do all this in an interactive, dynamic environment. + +!!Adding tests + +We all know that testing is good, but how do we actually test a web app ? Writing some basic tests is actually not difficult, since Zinc HTTP Components covers both the client and the server side with the same objects. + +Writing tests is creating objects, letting them interact and then asserting a number of conditions. Create a new subclass of ==TestCase==, ==MyFirstWebAppTests==, and add the following helper method. + +[[[ +MyFirstWebAppTests>>withServerDo: block + | server | + server := ZnServer on: 1700 + 10 atRandom. + [ + server start. + self assert: server isRunning & server isListening. + server delegate: MyFirstWebApp new. + block cull: server + ] ensure: [ server stop ] +]]] + +Since we will need a configured server instance with our web app as delegate for each of our tests, we move that logic into #withServerDo: and make sure the server is OK and properly stopped afterwards. Now we are ready for our first test. + + +[[[ +MyFirstWebAppTests>>testMainPage + self withServerDo: [ :server | + | client | + client := ZnClient new. + client url: server localUrl; addPath: #image. + client get. + self assert: client isSuccess. + self assert: (client entity contentType matches: ZnMimeType textHtml). + self assert: (client contents includesSubstring: 'Image'). + client close ] + +]]] + + +In ==testMainPage== we do a request for the main page, /image, and assert that the request is successful and contains HTML. Make sure the test is green by running it from the system browser (click the round icon in front of the method name in the fourth pane). + + +@@todo puc + +Let's try to write a test for the actual raw image being served. + +[[[ +MyFirstWebAppTests>>testDefaultImage + self withServerDo: [ :server | + | client | + client := ZnClient new. + client url: server localUrl; addPath: #image; queryAt: #raw put: #true. + client get. + self assert: client isSuccess. + self assert: (client entity contentType matches: 'image/*' asZnMimeType). + self assert: client entity equals: server delegate image. + client close ] +]]] + +Note how we can actually test for equality between the served image and the one inside our app object (the delegate). Run the test. + +Our final test will actually do an image upload and check if the served image did actually change to what we uploaded. Here we define the method ==image== that returns a new image. + +[[[ +MyFirstWebAppTests>>image + ^ ZnClient new + beOneShot; + get: 'http://zn.stfx.eu/zn/Hot-Air-Balloon.gif'; + entity + +MyFirstWebAppTests>>testUpload + self withServerDo: [ :server | + | image client | + image := self image. + client := ZnClient new. + client url: server localUrl; addPath: #image. + client addPart: (ZnMimePart fieldName: #file entity: image). + client post. + self assert: client isSuccess. + client resetEntity; queryAt: #raw put: #true. + client get. + self assert: client isSuccess. + self assert: client entity equals: image. + client close ] +]]] + +The HTTP client object is pretty powerful. It can do a correct multi-part form-data POST, just like a browser. Furthermore, once configured, it can be reused, like for the second GET request. + +!! Saving code to a repository + +If all is well, you now have a package called ==MyFirstWebApp== containing two classes, ==MyFirstWebApp== and ==MyFirstWebAppTests==. The first one should have 9 methods, the second 5. If you are unsure about your code, you can double check with the full listing at the end of this document. Our web app should now work as expected, and we have some tests to prove it. + +But our code currently only lives in our development image. Let's change that and move our code to a source code repository. For this we first have to define a Monticello package. Click on the package name in the first column of the browser and select the option 'Create an MC package'. Use the same name. + +@@todo Creating a monticello packag + +Pharo uses distributed source code management. Your code can live on your local file system, or it can live on a server. The main place for storing Pharo code is on SmalltalkHub http://www.smalltalkhub.com. Go over there and create yourself a new account. Once you have an account, create and register a new project called =='MyFirstWebApp'==. You can leave the public option checked, it means that you and others can download the code without credentials. Go to the project's page. + +@@todo MyFirstWebApp’s project page on SmalltalkHub + +On this page, select and copy the Monticello registration template (make sure to copy the whole contents, including the username and password parts). Now go back to Pharo and add a repository for your package (right-click on the package name, select Open... Add a repository). + +Select Smalltalkhub.com as repository type and overwrite the presented template with the one you just copied. It should look similar to + +[[[ +MCHttpRepository + location: 'http://www.smalltalkhub.com/mc/SvenVanCaekenberghe/MyFirstWebApp/mai + user: '' + password: '' +]]] + +Now before accepting, fill in your user(name) and password (between the single quotes), the ones you gave during registration on SmalltalkHub. + +Open the Monticello Browser to see what we have done. Find your package in the first column and your repository in the second one. + + +@@todo The Monticello Browser looking at our package and repository + +There should be a asterisk (==*==) in front of your package name, indicating that the package is dirty, that it has uncommitted changes. If not, force a change computation by clicking 'Changes' button. You should get a browser showing all the changes that you made. Since this is the first version, all your changes are additions. + +!!!!The Changes/Diff Browser for our package. + +OK, we're almost done. Go back to the Monticello Browser and click the 'Save' button (with your package and repository selected). Leave the version name, something like MyFirstWebApp-SvenVanCaekenberghe.1 alone, write a nice commit message in the second pane and press Accept to save your code to SmalltalkHub. + +!!!!Commiting to SmalltalkHub +When all goes well, you will see an upload progress bar and finally a version window that confirms the commit. You can close it later on. + +[[[ +Name: MyFirstWebApp-SvenVanCaekenberghe.1 + Author: SvenVanCaekenberghe + Time: 9 July 2013, 2:18:24.638381 pm + UUID: adad42a6-ff4c-41a4-a2a3-09f8cb29c902 + Ancestors: + First check in of our web app, following 'Building and deploying your first web a +]]] + +!!!Confirmation Version Window. + +If something goes wrong, you probably made a typo in your repository specification. You can edit it by right-clicking on it in the Monticello Browser and selecting ‘Edit repository info’. If a save fails, you will get a Version Window after some error message. Don’t close the Version Window. Your code now lives in your local package cache. Click the ‘Copy’ button and select your SmalltalkHub repository to try saving again. + +You can now browse back to Smalltalkhub.com to confirm that your code arrived there. + +!!!! Looking at our commit on SmalltalkHub. + +After a successful commit, it is a good idea to save your image. In any case, your package should now no longer be dirty, and there should be no more differences between the local version and the one on SmalltalkHub. + +!!Defining a project configuration + +Real software consists of several packages and will depend on extra external libraries and frameworks. In practice, software configuration management, including the management of dependencies and versions, is thus a necessity. + +To solve this problem, Pharo is using Metacello (there is a full chapter on the book Deep into Pharo). And although we don’t really need it for our small example, we are going to use it anyway. Of course, we will not go into details as this is a complex subject. + +To create a Metacello configuration, you define an object, what else did you expect ? First create a new package as well as a Metacello package called ==ConfigurationOfMyFirstWebApp==. Then go find the class ==MetacelloConfigTemplate==. You have to copy this class (right-click on the class name) and name it ==ConfigurationOfMyFirstWebApp== as well. Now move the copy to your new package by dragging it, or by editing the category field of the class definition. + +We are going to define three methods: one defining a baseline for our configuration, one defining concrete package versions for that baseline, and one declaring that version as the stable released version. Here is the code (if you would be working in Pharo 3.0 you will notice that ==MetacelloConfigTemplate== contains some extra template methods, remove any baseline or version related ones and overwrite ==stable:==) + +[[[ +ConfigurationOfMyFirstWebApp>>baseline1: spec + + spec for: #common do: [ + spec + blessing: #baseline; + repository: 'http://smalltalkhub.com/mc/SvenVanCaekenberghe/MyFirstWebApp/m + package: 'MyFirstWebApp' ] +ConfigurationOfMyFirstWebApp>>version1: spec + + spec for: #common do: [ + spec + blessing: #release; + package: 'MyFirstWebApp' with: 'MyFirstWebApp-SvenVanCaekenberghe.1' ] +ConfigurationOfMyFirstWebApp>>stable: spec + + spec for: #common version: '1' +]]] + + +You can test your configuration by trying to load it. + +[[[ +ConfigurationOfMyFirstWebApp load. +]]] + + +Of course, not much will happen since you already have the specified version loaded. For some feedback, make sure the Transcript is open and inspect the above expression. + + + +Now add your SmalltalkHub repository to the ConfigurationOfMyFirstWebApp Monticello package. Double-check the changes in the Monticello Browser, remember we copied a whole class. Now commit by saving to your SmalltalkHub repository. Use the web interface to verify that all went well. + + + +!!Running a real cloud server + + +So we created our first web app and tested it locally. We stored our source code in the SmalltalkHub repository and created a Metacello configuration for it. Now we need a real cloud server to run our web app. + +It used to be hard and expensive to get access to a real server permanently connected to the internet. Not any more: prices have comes down and operating cloud servers has become a much easier to use service. + +For this guide, we will be using Digital Ocean. The entry level server there, which is more than powerful enough for our experiment, costs just $5 a month. If you stop and remove the server after a couple of days, you will only pay cents. Go ahead and make yourself an account and register a credit card. + +!!!First part of the Create Droplet form + +A server instance is called a Droplet. Click the ‘Create Droplet ’ button and fill in the form. Pick a hostname, select the smallest size, pick a region close to you. As operating system image, we’ll be using a 32-bit Ubuntu Linux, version 13.04 x32. You can optionally use an SSH key pair to log in - it is a good idea, see How to Use SSH Keys with DigitalOcean Droplets - just skip this option for now if you are uncomfortable with it, it is not necessary for this tutorial. Finally click the ‘Create Droplet’ button. + + +!!! Second part of the Create Droplet form + +In less than a minute, your new server instance will be ready. Your root password will be emailed to you. If you look at your droplets, you should see your new server in the list. Click on it to see its details. + +!!! Looking at your Droplet + +The important step now is to get SSH command line access to your new server, preferably using a normal terminal. With the IP address from the control panel and the root password emailed to you, try to log in. +[[[ + $ ssh root@82.196.12.54 +]]] + +Your server is freshly installed and includes only the most essential core packages. Now we have to install Pharo on it. One easy way to do this is using the functionality offered by *http://get.pharo.org*. The following command will install a fresh Pharo 2.0 image together with all other files needed. + +[[[ + # curl get.pharo.org/20+vm | bash +]]] + +Make sure the VM\+image combination works by asking for the image version. + + +[[[ +# ./pharo Pharo.image printVersion +[version] 2.0 #20611 +]]] + +Let's quickly test the stock HTTP server that comes with Pharo, like we did in the third section of this guide. + +[[[ +# ./pharo Pharo.image eval --no-quit 'ZnServer startDefaultOn: 8080' +]]] + +This command will block. Now access your new HTTP server at *http://82.196.12.54:8080* after substituting your own IP address of course. You should see the Zinc HTTP Components welcome page. If this works, you can press ctrl-C in the terminal to end our test. + +!! Deploying for production + +We now have a running server. It can run Pharo too, but it is currently using a generic image. How do we get our code deployed ? To do this we use the Metacello configuration. But first, we are going to make a copy of the stock Pharo.image that we downloaded. We want to keep the original clean while we make changes to the copy. + +[[[ +# ./pharo Pharo.image save myfirstwebapp +]]] + +We now have a new image (and changes) file called myfirstwebapp.image (and myfirstwebapp.changes). Through the config command line option we can load our Metacello configuration. Before actually loading anything, we will ask for all available versions to verify that we can access the repository. + +[[[ +# ./pharo myfirstwebapp.image config http://www.smalltalkhub.com/mc/SvenVanCaeken + =============================================================================== + Notice: Available versions for ConfigurationOfMyFirstWebApp + =============================================================================== + 1 + 1-baseline + bleedingEdge + last + stable +]]] + +Since we have only one version, all the above are equivalent references to the same version. Now we will load and install the stable version. + +[[[ +# ./pharo myfirstwebapp.image config http://www.smalltalkhub.com/mc/SvenVanCaeken + =============================================================================== + Notice: Installing ConfigurationOfMyFirstWebApp stable + =============================================================================== +]]] + + +After loading all necessary code, the config option will also save our image so that it now permanently includes our code. Although we could try to write a (long) one line expression to start our web app in a server and pass it to the eval option, it is better to write a small script. Create a file called ‘run.st’ with the following contents + +[[[ +ZnServer defaultOn: 8080. + ZnServer default logToStandardOutput. + ZnServer default delegate + map: 'image' to: MyFirstWebApp new; + map: 'redirect-to-image' to: [ :request | ZnResponse redirect: 'image' ]; + map: '/' to: 'redirect-to-image'. +ZnServer default start. +]]] + +We added a little twist here: we changed the default root (==/==) handler to redirect to our new ==/image== web app. Test the startup script like this + +[[[ +# ./pharo myfirstwebapp.image run.st + 2013-07-10 11:46:58 660707 I Starting ZnManagingMultiThreadedServer HTTP port 808 + 2013-07-10 11:46:58 670019 D Initializing server socket + 2013-07-10 11:47:12 909356 D Executing request/response loop + 2013-07-10 11:47:12 909356 I Read a ZnRequest(GET /) + 2013-07-10 11:47:12 909356 T GET / 302 16B 0ms + 2013-07-10 11:47:12 909356 I Wrote a ZnResponse(302 Found text/plain;charset=utf- + 2013-07-10 11:47:12 909356 I Read a ZnRequest(GET /image) + 2013-07-10 11:47:12 909356 T GET /image 200 282B 0ms + 2013-07-10 11:47:12 909356 I Wrote a ZnResponse(200 OK text/html;charset=utf-8 28 + 2013-07-10 11:47:12 909356 I Read a ZnRequest(GET /image?raw=true) + 2013-07-10 11:47:12 909356 T GET /image?raw=true 200 18778B 82ms + 2013-07-10 11:47:12 909356 I Wrote a ZnResponse(200 OK image/png 18778B) +]]] + +Surf to the correct IP address and port to test you application. Note that ==/welcome==, ==/help== and ==/image== are still available too. Type ctrl-c to kill the server again. Now it is time to put the server in background, running for real. + +[[[ +# nohup ./pharo myfirstwebapp.image run.st & +]]] + +!!One more step + +Did you like the example so far ? Would you like to take one more step ? Here is a little extension, as an exercise. +Add an extra section at the bottom of the main page that shows a miniature version of the previous image. Initially, you can show an empty image. Here are a couple of hints. Read only as far as you need, try to figure it out yourself. + +!!!Hint 1 +You can scale a form object into another one using just one message taking a single argument. You can use the same classes that we used for parsing as a tool to generate PNG, JPEG or GIF images given a form. + +When you are done, save your code as a new version. Then update your configuration with a new, stable version. Finally, go to the server, update your image based on the configuration and restart the running vm\+image. + +!!!Hint 2 +Change the #html method referring to a new variant, ==/image?previous=true==, for the second image. Adjust ==handleGetRequest:== to look for that attribute. +Add a helper method ==pngImageEntityForForm:== and a ==previousImage== accessor. It is easy to create an empty, blank form as default. Call a ==updatePreviousImage== at the right spot in ==handlePostRequest:== and implement the necessary functionality there. + +!!!Hint 3 +If you found it difficult to find the right methods, have a look at the following ones: + +-Form>>scaledIntoFormOfSize: +-Form class>>extent:depth: +-PNGReadWriter>>nextPutImage: +-ByteArray class>>streamContents: +-ZnByteArrayEntity class>>with:type: + +!!Solution, part 1, new methods + +Here are 3 new methods that are part of the solution. + +[[[ +pngImageEntityForForm: form + ^ ZnByteArrayEntity + with: (ByteArray streamContents: [ :out | + (PNGReadWriter on: out) nextPutImage: form ]) + type: ZnMimeType imagePng + +previousImage + ^ previousImage ifNil: [ + | emptyForm | + emptyForm:= Form extent: 128 @ 128 depth: 8. + previousImage := self pngImageEntityForForm: emptyForm ] + +updatePreviousImage + | form scaled | + form := self form. + scaled := form scaledIntoFormOfSize: 128. + previousImage := self pngImageEntityForForm: scaled +]]] + +!!Solution, part 2, changed methods + +Here are the changes to 3 existing methods for the complete solution. +[[[ + html + ^ 'Image + +

Image

+ +
+
+

Change the image:

+ + +
+

Previous Image

+ + ' + + handleGetRequest: request + (request uri queryAt: #raw ifAbsent: [ nil ]) + ifNotNil: [ ^ ZnResponse ok: self image ]. + (request uri queryAt: #previous ifAbsent: [ nil ]) + ifNotNil: [ ^ ZnResponse ok: self previousImage ]. + ^ ZnResponse ok: (ZnEntity html: self html) + + handlePostRequest: request + | part newImage badRequest | + badRequest := [ ^ ZnResponse badRequest: request ]. + (request hasEntity + and: [ request contentType matches: ZnMimeType multiPartFormData ]) + ifFalse: badRequest. + part := request entity + partNamed: #file + ifNone: badRequest. + newImage := part entity. + (newImage notNil + and: [ newImage contentType matches: 'image/*' asZnMimeType ]) + ifFalse: badRequest. + [ self formForImageEntity: newImage ] + on: Error + do: badRequest. + self updatePreviousImage. + image := newImage. + ^ ZnResponse redirect: #image +]]] + +!!Solution, part 3, updated configuration +To update our configuration, add 1 method and change 1 method. + +[[[ +version2: spec + + spec for: #common do: [ + spec + blessing: #release; + package: 'MyFirstWebApp' with: 'MyFirstWebApp-SvenVanCaekenberghe.2' ] + stable: spec + + spec for: #common version: '2' +]]] + +Of course, you will have to substitute your name for the concrete version. + +!!Conclusion +Congratulations: you have now built and deployed your first web app with Pharo. Hopefully you are interested in learning more. From the Pharo website you should be able to find all the information you need. Don’t forget about the Pharo by Example book and the mailing lists. +This guide was an introduction to writing web applications using Pharo, touching on the fundamentals of HTTP. Like we mentioned in the introduction, there are a couple of high level frameworks that offer more extensive support for writing web applications. The three most important ones are Seaside, AIDAweb and Iliad. + +!!Listing +Here is the full code listing of the web app. You can also find the code, including the tests and the Metacello configuration, checked in to SmalltalkHub in my MyFirstWebApp project. A similar example is also included in Zinc HTTP Components itself, under the name ==ZnImageExampleDelegate==[Tests]. + +[[[ +Object subclass: #MyFirstWebApp + instanceVariableNames: '' + classVariableNames: '' + poolDictionaries: '' + category: 'MyFirstWebApp' +]]] + +[[[ +handleRequest: request + request uri path = #image + ifTrue: [ + request method = #GET + ifTrue: [ ^ self handleGetRequest: request ]. + request method = #POST + ifTrue: [ ^ self handlePostRequest: request ] ]. + ^ ZnResponse notFound: request uri +]]] + + +[[[ +value: request + ^ self handleRequest: request +]]] + +[[[ +handleGetRequest: request + ^ (request uri queryAt: #raw ifAbsent: [ nil ]) + ifNil: [ ZnResponse ok: (ZnEntity html: self html) ] + ifNotNil: [ ZnResponse ok: self image ] +]]] + +[[[ +handlePostRequest: request + | part newImage badRequest | + badRequest := [ ^ ZnResponse badRequest: request ]. + (request hasEntity + and: [ request contentType matches: ZnMimeType multiPartFormData ]) + ifFalse: badRequest. + part := request entity + partNamed: #file + ifNone: badRequest. + newImage := part entity. + (newImage notNil + and: [ newImage contentType matches: 'image/*' asZnMimeType ]) + ifFalse: badRequest. + [ self formForImageEntity: newImage ] + on: Error + do: badRequest. + image := newImage. + ^ ZnResponse redirect: #image +]]] + +[[[ +html + ^ 'Image + +

Image

+ +
+
+

Change the image:

+ + +
+ ' +]]] + +[[[ +downloadPharoLogo + ^ ZnClient new + beOneShot; + get: 'http://www.pharo-project.org/images/pharo.png'; + entity +]]] + +[[[ +image + ^ image ifNil: [ image := self downloadPharoLogo ] +]]] + + +[[[ +formForImageEntity: imageEntity + | imageType parserClassName parserClass parser | + imageType := imageEntity contentType sub. + parserClassName := imageType asUppercase, #ReadWriter. + parserClass := Smalltalk globals at: parserClassName asSymbol. + parser := parserClass on: imageEntity readStream. + ^ parser nextImage +]]] + +[[[ +form + ^ self formForImageEntity: self image +]]] + +% Local Variables: +% eval: (flyspell-mode -1) +% End: diff --git a/WebApp/figures/add-repo.png b/WebApp/figures/add-repo.png new file mode 100644 index 0000000..9b99266 Binary files /dev/null and b/WebApp/figures/add-repo.png differ diff --git a/WebApp/figures/breakpoint-1.png b/WebApp/figures/breakpoint-1.png new file mode 100644 index 0000000..aeca2e6 Binary files /dev/null and b/WebApp/figures/breakpoint-1.png differ diff --git a/WebApp/figures/breakpoint-2.png b/WebApp/figures/breakpoint-2.png new file mode 100644 index 0000000..0245dac Binary files /dev/null and b/WebApp/figures/breakpoint-2.png differ diff --git a/WebApp/figures/commit.png b/WebApp/figures/commit.png new file mode 100644 index 0000000..296337a Binary files /dev/null and b/WebApp/figures/commit.png differ diff --git a/WebApp/figures/create-droplet-1.png b/WebApp/figures/create-droplet-1.png new file mode 100644 index 0000000..191edd3 Binary files /dev/null and b/WebApp/figures/create-droplet-1.png differ diff --git a/WebApp/figures/create-droplet-2.png b/WebApp/figures/create-droplet-2.png new file mode 100644 index 0000000..1c09098 Binary files /dev/null and b/WebApp/figures/create-droplet-2.png differ diff --git a/WebApp/figures/create-mc-package.png b/WebApp/figures/create-mc-package.png new file mode 100644 index 0000000..3b4c628 Binary files /dev/null and b/WebApp/figures/create-mc-package.png differ diff --git a/WebApp/figures/diffs.png b/WebApp/figures/diffs.png new file mode 100644 index 0000000..5cd3886 Binary files /dev/null and b/WebApp/figures/diffs.png differ diff --git a/WebApp/figures/dnu-1.png b/WebApp/figures/dnu-1.png new file mode 100644 index 0000000..2b40364 Binary files /dev/null and b/WebApp/figures/dnu-1.png differ diff --git a/WebApp/figures/dnu.png b/WebApp/figures/dnu.png new file mode 100644 index 0000000..2b40364 Binary files /dev/null and b/WebApp/figures/dnu.png differ diff --git a/WebApp/figures/first-code.png b/WebApp/figures/first-code.png new file mode 100644 index 0000000..4d06739 Binary files /dev/null and b/WebApp/figures/first-code.png differ diff --git a/WebApp/figures/image-web-app.png b/WebApp/figures/image-web-app.png new file mode 100644 index 0000000..1f8a95e Binary files /dev/null and b/WebApp/figures/image-web-app.png differ diff --git a/WebApp/figures/live-change.png b/WebApp/figures/live-change.png new file mode 100644 index 0000000..04de7c2 Binary files /dev/null and b/WebApp/figures/live-change.png differ diff --git a/WebApp/figures/pharo-in-action.png b/WebApp/figures/pharo-in-action.png new file mode 100644 index 0000000..6323f55 Binary files /dev/null and b/WebApp/figures/pharo-in-action.png differ diff --git a/WebSockets/WebSockets.pier b/WebSockets/WebSockets.pier new file mode 100644 index 0000000..2431023 --- /dev/null +++ b/WebSockets/WebSockets.pier @@ -0,0 +1,264 @@ +!WebSockets +Sven Van Caekenberghe + +September 2012 - Updated January 2013 +(This is a draft) + +The WebSocket protocol defines a full-duplex single socket connection over which messages can be sent between a client and a server. It simplifies much of the complexity around bi-directional web communication and connection management. WebSocket represents the next evolutionary step in web communication compared to Comet and Ajax. + +!!An Introduction to WebSockets + +HTTP, one of the main technologies of the internet, defines a communication protocol between a client and a server where the initiative of the communication lies with the client and each interaction consists of a client request and a server response. When correctly implemented and used, HTTP is enormously scaleable and very flexible. + +With the arrival of advanced Web applications mimicking regular desktop applications with rich user interfaces, as well as mobile Web applications, it became clear that HTTP was not suitable or not a great fit for two use cases: + +when the server wants to take the initiative and send the client a message +when the client wants to send (many) (possibly asynchronous) short messages with little overhead +In the HTTP protocol, the server cannot take the initiative to send a message, the only workaround is for the client to do some form of polling. For short messages, the HTTP protocol adds quite a lot of overhead in the form of meta data headers. For many applications, the response (and the delay waiting for it) are not needed. Previously, Comet and Ajax were used as (partial) solutions to these use cases. + +The WebSocket protocol defines a reliable communication channel between two equal parties, typically, but not necessarily, a Web client and a Web server, over which asynchronous messages can be send with very little overhead. Messages can be any String or ByteArray. Overhead is just a couple of bytes. There is no such thing as a direct reply or a synchronous confirmation. + +Using WebSockets, a server can notify a client instantly of interesting events, and clients can quickly send small notifications to a server, possibly multiplexing many virtual communications channels over a single network socket. + +!!The WebSocket Protocol + +Zinc WebSockets implements RFC 6455 http://tools.ietf.org/html/rfc6455, not any of the previous development versions. For an introduction to, both http://en.wikipedia.org/wiki/WebSocket and http://www.websocket.org are good starting points. + +As a protocol, WebSocket starts with an initial setup handshake that is based on HTTP. The initiative for setting up a WebSocket lies with the client, who is sending a so called connection upgrade request. The upgrade request contains a couple of special HTTP headers. The server begins as a regular HTTP server accepting the connection upgrade request. When the request is conform the specifications, a 101 switching protocols response is sent. This response also contains a couple of special HTTP headers. From that point on, the HTTP conversation over the network socket stops and the WebSocket protocol begins. + +WebSocket messages consist of one or more frames with minimal encoding. Behind the scenes, a number of control frames are used to properly close the WebSocket and to manage keeping alive the connection using ping and pong frames. + +!!Source Code + +The code implementing Zinc WebSockets resides in a single package called 'Zinc-WebSocket-Core' in the Zinc HTTP Components repositories. There is also an accompanying 'Zinc-WebSocket-Tests' package containing the unit tests. The ==ConfigurationOfZincHTTPComponents== has a group called =='WebSocket'== that you can load separately. + +!!!Using Client Side WebSockets + +An endpoint for a WebSocket is specified using a URL + +[[[ +ws://www.example.com:8088/my-app +]]] + +Two new schemes are defined, ==ws://== for regular WebSockets and ==wss://== for the secure (TLS/SSL) variant. The ==host:port== and path specification should be familiar. + +Zinc WebSockets supports the usage of client side WebSockets of both the regular and secure variants (the secure variant requires Zodiac TLS/SSL). The API is really simple, once you open the socket, you use ==sendMessage:== and ==readMessage:== and finally ==close==. + +Here is a client side example taking to a public echo service: + +[[[ +| webSocket | +webSocket := ZnWebSocket to: 'ws://echo.websocket.org'. +[ webSocket + sendMessage: 'Pharo Smalltalk using Zinc WebSockets !'; + readMessage ] ensure: [ webSocket close ]. +]]] + +Note that ==readMessage:== is blocking. It always returns a complete ==String== or ==ByteArray==, possible assembled out of multiple frames. Inside ==readMessage:== control frames will be handled automagically. Reading and sending are completely separate and independent. + +For sending very large messages, there are ==sendTextFrames:== and ==sendByteFrames:== that take a collection of ==Strings== or ==ByteArrays== to be sent as different frames of the same message. At the other end, these will be joined together and seen as a single message. + +In any non-trivial application, you will have to add your own encoding and decoding to messages. In many cases, JSON will be the obvious choice as the client end is often JavaScript. A modern, standalone JSON parser and writer is NeoJSON. + +To use secure web sockets, just use the proper URL scheme ==wss://== as in the following example: + +[[[ +| webSocket | +webSocket := ZnWebSocket to: 'wss://echo.websocket.org'. +[ webSocket + sendMessage: 'Pharo Smalltalk using Zinc WebSockets & Zodiac !'; + readMessage ] ensure: [ webSocket close ]. +]]] + +Of course, your image has to contain Zodiac and your VM needs access to the proper plugin. That should not be a problem with the latest Pharo releases. + +!!Using Server Side WebSockets + +Since the WebSocket protocol starts off as HTTP, it is logical that a ZnServer with a special delegate is the starting point. ==ZnWebSocketDelegate== implements the standard ==handleRequest:== to check if the incoming request is a valid WebSocket connection upgrade request. If so, the matching 101 switching protocols response is constructed and sent. From that moment on, the network socket stream is handed over to a new, server side ==ZnWebSocket== object. + + ==ZnWebSocketDelegate== has two properties. An optional prefix implements a specific path, like ==/my-ws-app==. The required handler is any object implementing ==value:== with the new web socket as argument. + +Let's implement the echo service that we connected to as a client in the previous subsection. In essence, we should go in a loop, reading a message and sending it back. Here is the code: + +[[[ +ZnServer startDefaultOn: 1701. +ZnServer default delegate: (ZnWebSocketDelegate handler: + [ :webSocket | + [ | message | + message := webSocket readMessage. + webSocket sendMessage: message ] repeat ]). +]]] + +We start a default server on port 1701 and replace its delegate with a ==ZnWebSocketDelegate==. The ==ZnWebSocketDelegate== will pass each correct web socket request on to its handler. In this example, a block is used as handler. The handler is given a new connected ==ZnWebSocket== instance. For the echo service, we go into a repeat loop, reading a message and sending it back. + +Finally, you can stop the server using + +[[[ +ZnServer stopDefault. +]]] + +Although the code above works, it will eventually encounter two ==NetworkErrors==: + +[[[ +ConnectionTimedOut +ConnectionClosed (or its more specific subclass ZnWebSocketClosed) +]]] + +The ==readMessage== call blocks on the socket stream waiting for input until its timeout expires, which will be signaled with a ConnectionTimedOut exception. In most applications, you should just keep on reading, essentially ignoring the timeout for an infinite wait on incoming messages. + +This behavior is implemented in the ==ZnWebSocket>>runWith:== convenience method: it enters a loop reading messages and passing them to a block, continuing on timeouts. This simplifies our example: + +[[[ +ZnServer startDefaultOn: 1701. +ZnServer default delegate: (ZnWebSocketDelegate handler: + [ :webSocket | + webSocket runWith: [ :message | + message := webSocket readMessage. + webSocket sendMessage: message ] ]). +]]] + +That leaves us with the problem of ==ConnectionClosed==. This exception can occur at the lowest level when the underlying network connection closes unexpectedly, or at the WebSocket protocol level when the other end sends a close frame. In either case we have to deal with it as a server. In our trivial echo example, we can catch and ignore any ConnectionClosed exception. + +There is a handy shortcut method on the class side of ==ZnWebSocket== that helps to quickly set up a server implementing a WebSocket service. + +[[[ +ZnWebSocket + startServerOn: 8080 + do: [ :webSocket | + [ + webSocket runWith: [ :message | + message := webSocket readMessage. + webSocket sendMessage: message ] ] + on: ConnectionClosed + do: [ self crLog: 'Ignoring connection close, done' ] ]. +]]] + +Don't forget to inspect the above code so that you have a reference to the server to close it, as this will not be the default server. + +Although using a block as handler is convenient, for non-trivial examples a regular object implementing ==value:== will probably be better. You can find such an implementation in ==ZnWebSocketEchoHandler==. + +The current process (thread) as spawned by the server can be used freely by the handler code, for as long as the web socket connection lasts. The responsibility for closing the connection lies with the handler, although a close from the other side will be handled correctly. + +To test our echo service, you could connect to it using a client side web socket, like we did in the previous subsection. This is what the unit test ==ZnWebSocketTests>>testEcho== does. Another solution is to run some JavaScript code in a web browser. You can find the necessary HTML page containing JavaScript code invoking the echo service on the class side of ==ZnWebSocketEchoHandler==. The following setup will serve this code: + +[[[ +ZnServer startDefaultOn: 1701. +ZnServer default logToTranscript. +ZnServer default delegate + map: 'ws-echo-client-remote' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtmlRemote) ]; + map: 'ws-echo-client' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtml) ]; + map: 'ws-echo' + to: (ZnWebSocketDelegate map: 'ws-echo' to: ZnWebSocketEchoHandler new). +]]] + +Now, you can try the following URLs: + +[[[ +http://localhost:1701/ws-echo-client-remote +http://localhost:1701/ws-echo-client +]]] + +The first one will connect to ==ws://echo.websocket.org== as a reference, the second one will connect to our implementation at ==ws://localhost:1701/ws-echo==. + +Another simple example is available in ==ZnWebSocketStatusHandler== where a couple of Smalltalk image statistics are emitted every second for an efficient live view in your browser. In this scenario, the server accepts each incoming web socket connection and starts streaming to it, not interested in any incoming messages. Here is the core loop: + +[[[ +ZnWebSocketStatusHandler>>value: webSocket + [ + self crLog: 'Started status streaming'. + [ + webSocket sendMessage: self status. + 1 second asDelay wait. + webSocket isConnected ] whileTrue ] + on: ConnectionClosed + do: [ self crLog: 'Ignoring connection close' ]. + self crLog: 'Stopping status streaming' +]]] + +The last example, ==ZnWebSocketChatroomHandler==, implements the core logic of a chatroom: clients can send messages to the server who distributes them to all connected clients. In this case, the handler has to manage a collection of all connected client web sockets. Here is the core loop: + +[[[ +ZnWebSocketChatroomHandler>>value: webSocket + [ + self register: webSocket. + webSocket runWith: [ :message | + self crLog: 'Received message: ', message printString. + self distributeMessage: message ] ] + on: ConnectionClosed + do: [ + self crLog: 'Connection close, cleaning up'. + self unregister: webSocket ] +]]] + +Distributing the message is as simple as iterating over each client (ignoring some details): + +[[[ +ZnWebSocketChatroomHandler>>distributeMessage: message + clientWebSockets do: [ :each | + each sendMessage: message ]. +]]] + +Here is code to setup all examples: + +[[[ +ZnServer startDefaultOn: 1701. +ZnServer default logToTranscript. +ZnServer default delegate + map: 'ws-echo-client-remote' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtmlRemote) ]; + map: 'ws-echo-client' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtml) ]; + map: 'ws-echo' + to: (ZnWebSocketDelegate map: 'ws-echo' to: ZnWebSocketEchoHandler new); + map: 'ws-chatroom-client' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketChatroomHandler clientHtml) ]; + map: 'ws-chatroom' + to: (ZnWebSocketDelegate map: 'ws-chatroom' to: ZnWebSocketChatroomHandler new); + map: 'ws-status-client' + to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketStatusHandler clientHtml) ]; + map: 'ws-status' + to: (ZnWebSocketDelegate map: 'ws-status' to: ZnWebSocketStatusHandler new). +]]] + +Visit any of the following URLs: + +[[[ +http://localhost:1701/ws-echo-client +http://localhost:1701/ws-status-client +http://localhost:1701/ws-chat-client +]]] + +Inside your Pharo image, you can also send chat messages, much like a moderator: + +[[[ +(ZnServer default delegate prefixMap at: 'ws-chatroom') + handler distributeMessage: 'moderator>>No trolling please!'. +]]] + +!!A Quick Tour of the Implementation + +All code resides in the =='Zinc-WebSocket-Core'== package. The wire level protocol, the encoding and decoding of frames can be found in ==ZnWebSocketFrame==. The key methods are ==writeOn:== and ==readFrom:== as well as the instance creation protocol. Together with the testing protocol and #printOn: these should give enough information to understand the implementation. + + ==ZnWebSocket== implements the protocol above frames, either from a server or a client perspective. The key methods are ==readMessage== and ==readFrame==, sending is quite simple. Client side setup can be found on the class side of ==ZnWebSocket==. Server side handling of the setup is implemented in ==ZnWebSocketDelegate==. + +Two exceptions, ==ZnWebSocketFailed== and ==ZnWebSocketClosed== and a shared ==ZnWebSocketUtils== class round out the core code. + +!!Live Demo + +There is a live demo available with the basic Zinc-WebSocket demos: echo, status & chatroom. + +[[[ +http://websocket.stfx.eu +]]] + +Have a look at ==ZnWebSocketDelegate class>>installExamplesInServer:== as a starting point to learn how this demo was set up. + +Setting up a production demo is complicated by the fact that most proxies and load balancers, most notable market leader Apache, do not (yet) deal correctly with the WebSocket protocol. It is thus easiest to organize things so that your client talk directly to your Smalltalk image. + +The implementation of Zinc WebSockets as an add-on to Zinc HTTP Components was made possible in part through financial backing by Andy Burnett of Knowinnovation Inc. and ESUG. + +!! Conclusion + +WebSockets integrate smoothly with Zinc to form another part of the Pharo web stack. \ No newline at end of file