enable third-party generators? (was: create chromeapp generator)

[damondouglas]

Is this an acceptable addition?

After completing #167, I envision the following:

$ stagehand client chrome
...
Available commands:
  simple    Generate simple chrome application.
  polymer   Generate polymer chrome application.

I would like to accomplish this via extending existing web app and polymer generators so that as these change, the chrome app generators change.

[sethladd]

Thanks for the suggestion!

@devoncarew I think it's time to moving to PHASE TWO... how to enable third-party generators. Curious if you have any thoughts on how we might do this.

[Andersmholmgren]

\o/
On Wed, 14 Jan 2015 at 11:41 am, Seth Ladd [email protected] wrote:

Thanks for the suggestion!

@devoncarew https://github.com/devoncarew I think it's time to moving
to PHASE TWO... how to enable third-party generators. Curious if you have
any thoughts on how we might do this.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-69977842.

[damondouglas]

@sethladd does enabling third-party generators mean using generators that live outside this repo? If so, would it be acceptable to pass a git url via a command-line argument? Then, stagehand would clone the repo into the target directory and remove the .git directory. If this works, then I wonder if we could possibly move all the generators into separate git repos for ease of maintenance.

[sethladd]

Yes, third-party generators means generators that live outside this repo.

I was wondering if we could scan installed pub packages and find locally
installed packages that contain templates.
On Wed Jan 14 2015 at 3:07:09 PM Damon Douglas [email protected]
wrote:

@sethladd https://github.com/sethladd does enabling third-party
generators mean using generators that live outside this repo? If so, would
it be acceptable to pass a git url via a command-line argument? Then,
stagehand would clone the repo into the target directory and remove the
.git directory. If this works, then I wonder if we could possibly move all
the generators into separate git repos for ease of maintenance.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-70011092.

[devoncarew]

OK, sounds like we want to think about 3rd party generators. I think we're discussing two options:

templates live on github Essentially what @damondouglas suggested. A template is a github repo, and the template id is the github repo name. So doing something like stagehand damondouglas/chrome-app would look at http://github.com/damondouglas/chrome-app for a repo, and if it found one, download and instantiate it. The repo itself would probably be a fully formed project (with a pubspec, lib/, tests/, ... directories), and probably some mustaches for stagehand to inject some user varaibles ({{projectname}}, {{username}}, ...).

templates live in pub packages What @sethladd suggested. We introspect on packages installed on the system (harder than it sounds...), looking for packages that contain template data in some well known location. I'm not sure what the format of the templates looks like in the packages. We've had to do some unfortunate build steps to copy our template resources into source code, so they're visible to us at runtime. I wouldn't want other template authors to have to also do this.

The first option isn't dissimilar to how bower works - an explicit dependency on github paths. It would make it very easy to de-centralize how stagehand works - anybody could add a new template without the stagehand author's permissions or review. But it would still let us ship a small set of curated samples built-in. So, the 5-7 samples we want to have available, which would just be a list of 5-7 repos containing those templates.

[sethladd]

A couple of questions with templates from GitHub:

How do we run the tool if the developer is offline?

How do we list all available templates?

[Andersmholmgren]

Heat about using pub global activate as a way to install Templates.

Pub itself supports path, hosted, git as three ways to located packages. Would be nice if stagehand was consistent and supported similar methods

[trentgrover-wf]

+100 on 3rd party generators

For my 2 cents, I like the idea of the package source being a fully formed project (as @devoncarew noted in his first option), but I'd like to see path, hosted, and git package locations supported as the initial source.

From that point, I'd suggest following a pattern something like this:

• install stagehand: pub global activate stagehand
• install a custom template (requires online connection for hosted / git packages): stagehand install [email protected]:whatever/myTemplate.git
• instantiate a new project based on the template (works offline because the install step cached it): stagehand myTemplate
• update the locally cached template if necessary: stagehand update myTemplate
• perhaps update all locally cached templates: stagehand update

Of course, once that first bit is in place, the very next ask would be for a simple way to prompt user for some more fine grained config options (a la yeoman's askFor methods).

[sethladd]

After chatting with @devoncarew, we'd like to open the request for a design doc and ultimately an implementation. We both would like to see stagehand facilitate third-party templates. If you're interested in helping make this happen, please feel free to jump in!

Also, what kinds of questions would we have to ask the user? What would the template need to know from the user?

[damondouglas]

@sethladd , @devoncarew , first extremely rough draft finished. I confess I had no idea what a design doc really was and googled it. https://docs.google.com/document/d/1eu6Acdjhwrt5LhyyrE9W0AAZhTKy0R5SqM3I7Wd5fcM/edit#

[sethladd]

Thanks! How can we do this without modifying any code?

Have you looked into Isolates as a way to dynamically load code?

[kunaldeo]

@sethladd thanks a lot. Using isolates here makes a lot more sense than anything else. I will be pushing a implementation which doesn't change the source of stagehand or third party template :) .

[kunaldeo]

To tackle Issue # 1 We can add stagehand as the dependency in the pubspec.yaml of the third party package.Fixed please see https://github.com/kunaldeo/stagehand/tree/third-party-templates .

Regarding Issue # 3 I am trying to use LibraryMirror to import the new types. If we don't import the new types mentioned in the sender isolate we get SegFault from VM. Please see this bug https://code.google.com/p/dart/issues/detail?id=15983 .

New Issue: When running as global package we get the following error:
Uncaught Error: Failure getting http://localhost:50393/stagehandData/isloader.dart: 404 Not Found
Looks like installed/compiled packages have limited file permissions :(. It works fine when it is run from source. This basically means isolates are not available from global packages.
All these issues are fixed. plugins package solve this problem by using isolates on the template side.

[damondouglas]

@sethladd , thank you for your recent suggestions to use grinder in the design document. Do you think the document is ready to post maybe on Google+ Dart community for feedback?

[sethladd]

Sorry for the delay. I'm a bit confused, though. What's the relationship between @damondouglas and @kunaldeo 's work?

[kunaldeo]

@sethladd No direct relationship. I just wanted to do a simple implementation of it. While it works I am not able to load isolates when running as package.

[sethladd]

@kunaldeo please leave a comment on https://code.google.com/p/dart/issues/detail?id=15983 so our engineers can know who this bug affects.

You won't be able to share a Type between two isolates loaded via spawnUri. You'll instead have to create a message-passing API and communicate with the other isolate-loaded template.

[damondouglas]

@sethladd sorry for the confusion. @kunaldeo , I gave you edit access to the design document. Maybe you and I can polish it up a bit and query feedback from the Dart community. Then, maybe if you still like me after that :), we can work together on a repo to push request.

[kunaldeo]

Sure @damondouglas. But we have to find what is technically possible with dart. I have filed a bug regarding not able to load isolate using Isolate.spawnUri when running from a globally installed pub package, https://code.google.com/p/dart/issues/detail?id=22436

[kunaldeo]

Thanks @sethladd for asking http://stackoverflow.com/questions/28549364/how-can-i-load-code-from-another-dart-package-known-only-at-runtime?stw=2 . plugins looks like the answer https://github.com/samrg472/plugins.dart . I am modifying my branch to use this.

[kunaldeo]

https://github.com/google/stagehand/pull/202 Includes my implementation of third party templates/plugins . @damondouglas nice work on the document 👍 My implementation does not support all the stuff mentioned in the document. Current implementation clones the repo to a directory called stagehandData then based on the plugins installed we add it to the generators . After that point stagehand takes care of it all. I like the idea of a grinder step which can build compatible third party templates. Or better yet provide a stagehand command line option which provides an option to generate third party template package. Sorry I couldn't modify the document. I am bad at editing docs.

Thanks @trentgrover-wf, implementation works exactly how you are proposing. Update command is still in works.

[sethladd]

w00t! Thanks very much for the patch. We'll take a look.

[devoncarew]

After noodling it over with a Seth, re-visiting the design ideas in this issue, and looking over @kunaldeo's PR (#204, thanks!), here's the direction we're thinking of going:

• Stagehand templates are just plain pub packages. They're easy to maintain; when under development they can be run like regular Dart apps. They're easy to version; they can be pushed up to pub once a new version is ready.
• They contain a small amount of additional metadata, either in the pubspec file or in a sidecar metadata file. This metadata will contain things like required user vars, optional user vars, and things to print out after the template has been created. So, the templates are entirely declarative.
• Users will install templates via 'pub global activate', just like they do for regular Dart CLI applications.
• Stagehand will scan the pubcache to know what templates are available. Stagehand's built in templates are just packages that it depends on and that get installed with the main tool via 'pub global activate stagehand'. i.e., they don't need any special treatment.
• We can do away with the generate-the-template-into-dart-source-code thing.

I think this will be a fairly small delta from the current version, will mean that stagehand can continue to ship with a small list of curated dart templates, but can easily support additional third-party templates.

[Andersmholmgren]

Makes sense to me. Leverage pub, make the simple things simple and leave
the door open to fancier stuff in the future if there is demand

On Sat, 14 Mar 2015 at 10:55 Devon Carew [email protected] wrote:

After noodling it over with a Seth, re-visiting the design ideas in this
issue, and looking over @kunaldeo https://github.com/kunaldeo's PR (#204
https://github.com/google/stagehand/pull/204, thanks!), here's the
direction we're thinking of going:

• Stagehand templates are just plain pub packages. They're easy to
  maintain; when under development they can be run like regular Dart apps.
  They're easy to version; they can be pushed up to pub once a new version is
  ready.
• They contain a small amount of additional metadata, either in the
  pubspec file or in a sidecar metadata file. This metadata will contain
  things like required user vars, optional user vars, and things to print out
  after the template has been created. So, the templates are entirely
  declarative.
• Users will install templates via 'pub global activate', just like
  they do for regular Dart CLI applications.
• Stagehand will scan the pubcache to know what templates are
  available. Stagehand's built in templates are just packages that it depends
  on and that get installed with the main tool via 'pub global activate
  stagehand'. i.e., they don't need any special treatment.
• We can do away with the generate-the-template-into-dart-source-code
  thing.

I think this will be a fairly small delta from the current version, will
mean that stagehand can continue to ship with a small list of curated dart
templates, but can easily support additional third-party templates.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-79539159.

[kunaldeo]

Thanks a lot for looking at the PR. I think git based approach will have the following advantages :

• As the dart ecosystem grows (like Python/Ruby) having templates as regular cli app on pub will uneven the pub repository. If number of templates are small its ok, but as the number grows you may find more templates in search results than anything else. Using git based approach we can keep the templates away from pub.
• Storing all templates at one location gives us easy maintenance as they are just git packages. For example they can all be updated at once.
• Since we are using isolates/plugins, templates can themselves execute some code based on the instructions from stagehand. While I have not demonstrated in current samples (See https://github.com/google/stagehand/pull/204), it cenrtainly opens some interesting possiblities. For example, I can pass on some arguments and data via stagehand to the template to only build the main menu and use the so and so firebase url. The stagehand template can then just generate the main menu, verify if the firebase url is valid and use it while building the project.

If this is an idea we are still toying with then I can build a template that demonstrate why loading code is useful.

[sethladd]

You bring up an interesting point... would we ever need to run pre or post
template hooks?

On Fri, Mar 13, 2015 at 6:39 PM Kunal Deo [email protected] wrote:

Thanks a lot for looking at the PR. I think git based approach will have
the following advantages :

• As the dart ecosystem grows (like Python/Ruby) having templates as
  regular cli app on pub will uneven the pub repository. If number of
  templates are small its ok, but as the number grows you may find more
  templates in search results than anything else. Using git based approach we
  can keep the templates away from pub.
• Storing all templates at one location gives us easy maintenance as
  they are just git packages. For example they can all be updated at once.
• Since we are using isolates/plugins, templates can themselves
  execute some code based on the instructions from stagehand. While I have
  not demonstrated in current samples (See First implementation of third party templates #204
  https://github.com/google/stagehand/pull/204), it cenrtainly opens
  some interesting possiblities. For example, I can pass on some arguments
  and data via stagehand to the template to only build the main menu and use
  the so and so firebase url. The stagehand template can then just generate
  the main menu, verify if the firebase url is valid and use it while
  building the project.

If this is an idea we are still toying with then I can build a template
that demonstrate why loading code is useful.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-79663776.

[kunaldeo]

We may need to run it, for example I have a polymer template which scaffolds a basic CRUD app using a firebase URL hooks can help template setup few stuff:

• Validate the firebase/db resource
• Build necessary resources/code/configuration for the CRUD app
• Pull external resources in the project

What I meant is having template hooks will allow us do things that can help the developer get started quickly. In Visual Studio 2013/Django you can create a new MVC project point it to a DB and it can generate a fairly good looking app quickly for you. I want something like this. Templates without intelligence need no integration to anything, but when you add intelligence to it it can be really useful.

Adding the hooks will be the template developer's responsibility. So if someone just wants to build a plain old template they can go with it, but if a really creative developer comes along he will have the opportunity to build a really dynamic stagehand template.

[sethladd]

Can we use pub to distribute and update templates, and still use plugins to allow for dynamic templates? That is, it doesn't look like Devon's ideas prohibit more dynamic templates.

[sethladd]

One of the first customers for this can be polymer.dart, to generate new polymer elements.

[sethladd]

Agreed there are cases where a template might have to do something
dynamically.

I'd like to suggest that we separate the feature requests into two:

1. Install, update, and run a third-party template.
2. Templates that can execute logic when then run.

I think we can and should do both.

On Fri, Mar 13, 2015 at 10:46 PM Kunal Deo [email protected] wrote:

We may need to run it, for example I have a polymer template which
scaffolds a basic CRUD app using a firebase URL hooks can help template
setup few stuff:

• Validate the firebase/db resource
• Build necessary resources/code/configuration for the CRUD app
• Pull external resources in the project

What I meant is having template hooks will allow us do things that can
help the developer get started quickly. In Visual Studio 2013 you can
create a new MVC project point it to a DB and it can generate a fairly good
looking app quickly for you. I want something like this. Templates without
intelligence need no integration to anything, but when you add intelligence
to it it can be really useful.

Adding the hooks will be the template developer's responsibility. So if
someone just wants to build a plain old template they can go with it, but
if a really creative developer comes along he will have the opportunity to
build a really dynamic stagehand template.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-79944130.

[Andersmholmgren]

I agree. If only templates that need run code incur the complexity
associated with it then that is a good thing imo
On Sun, 15 Mar 2015 at 10:44 am, Seth Ladd [email protected] wrote:

Agreed there are cases where a template might have to do something
dynamically.

I'd like to suggest that we separate the feature requests into two:

1. Install, update, and run a third-party template.
2. Templates that can execute logic when then run.

I think we can and should do both.

On Fri, Mar 13, 2015 at 10:46 PM Kunal Deo [email protected]
wrote:

We may need to run it, for example I have a polymer template which
scaffolds a basic CRUD app using a firebase URL hooks can help template
setup few stuff:

• Validate the firebase/db resource
• Build necessary resources/code/configuration for the CRUD app
• Pull external resources in the project

What I meant is having template hooks will allow us do things that can
help the developer get started quickly. In Visual Studio 2013 you can
create a new MVC project point it to a DB and it can generate a fairly
good
looking app quickly for you. I want something like this. Templates
without
intelligence need no integration to anything, but when you add
intelligence
to it it can be really useful.

Adding the hooks will be the template developer's responsibility. So if
someone just wants to build a plain old template they can go with it, but
if a really creative developer comes along he will have the opportunity
to
build a really dynamic stagehand template.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-79944130.

—
Reply to this email directly or view it on GitHub
https://github.com/google/stagehand/issues/192#issuecomment-80762497.

[kunaldeo]

About distributing templates using pub.dartlang.org . Templates itself are pub packages but in my opinion pub may not be the correct way to distribute templates. If you look at other package distribution system like PIP (Python), gem (Ruby), CPAN (perl) or nuget (dot net), none of them hosts templates in general. You can find some templates here and there but they dont serve as the mechanism to provide project templates.

Again, I am saying this betting on the fact that stagehand templates ecosystem will grow exponentially (because of developer productivity) having templates on pub will hinder the discoverability. Besides pub is better suited to host code that we can add to our projects or can be run stand alone. If we list the templates separately we can have our own categories like polymer/dynamic/simple/firebase/crud and have various filter system like polymer and firebase and show people what they are looking for without even knowing the template name.

Furthermore, an approach like this can help projects like DartVoid (by @andreasolund) and IDEs where they will not have to reinvent the wheel again and again.

[kasperpeulen]

I very satisfied with cookiecutter. With is a python command line tool, but also works for dart projects, and can generate templates from github repos. For example:
https://github.com/kasperpeulen/ng2-starter

But I also use it to generate specific boilerplate inside a project:
https://github.com/kasperpeulen/ng2-component

Maybe the third-party generator for stagehand can lend some inspiration from cookiecutter?

[sethladd]

Maybe cookiecutter can be ported to Dart? :)

[kasperpeulen]

@sethladd I started working on a port: https://github.com/kasperpeulen/cookiecutter (not finished)

I'm not sure how you envision this. In theory, after I ported it, I could maybe customize it a bit, so that stagehand could use it as a 3th party generator (as a dependency).

Or, it could also just be an alternative to stagehand, for generating dart projects.

[sethladd]

@kasperpeulen we could swap out the guts of stagehand for cookiecutter.

stagehand has two nice properties:

• It's used by all the major IDE/editors to generate new projects
• It has a machine-readable "interface" from the command line, making it easy to embed in other tools

If we can keep stagehand's machine-readable interface, then cookiecutter could become stagehand v2

How does that sound?

[kasperpeulen]

sounds great @sethladd

I will first try to finish the port of cookiecutter. I have not much time this week, but I expect to finish it next week.

I'm not totally sure what you mean with machine readable, and if I would need to add much to cookiecutter for that. But maybe better to come back for that once the port is finished.

[luisvt]

my 2 cents:

to add a yeoman generator you use npm, for example:

npm install -g generator-angular

furthermore all generators fallow the naming convention of adding generator- at the beginning of the name.

So basically what we need is just add something to stagehand to execute other global packages that begins with generator-. In other words, what we would need would be:

pub global activate generator-angular

and to run it, we would do:

stagehand angular

then internally stagehand would only need to do:

Process.start('generator-$name', [args]);

or something similar.

[felangel]

fwiw I’ve been working on mason which allows you to create your own custom templates using mustache which can be hosted either locally or on git.