Introducing imageflow_tool

Introducing imageflow_tool

TLDR; run imageflow_tool examples --generate and start opening issues on GitHub about what's broke or you think we should change.

One of Imageflow's key design goals is a compete and strict JSON interface which is unified and consistent across the library, server, and command-line form. I still haven't written all the error responses, but I think this technical preview of imageflow_tool will give you a good look at the format I'm aiming for with the universal JSON API. This release doesn't include an RIAPI or ImageResizer compatibility layer.

Ways of accessing the universal JSON API today

• With imageflow_tool via imageflow_tool v0.1/build --json request.json
• With libimageflow via imageflow_job_send_json(struct imageflow_context* context, struct imageflow_job* job, char const* method, uint8_t const* json_buffer, size_t json_buffer_size)
• With libimageflow-rb via Imageflow::Context.send_json(method:, data:) (this Ruby gem is not yet well factored or documented for reuse, but is also quite simple)

And later

• With imageflow_server via http://host/imageflow/v0.1/build
• With community-provided language bindings (official support is for Ruby and eventually Node, but @SamuelEnglard is working on C# bindings.

We don't have a Rust server ready yet (that requires far more security testing and coverage), but imageflow_server will implement essentially the same JSON API. Minor differences for request signatures and authorization/authentication may be required, but can probably be handled via HTTP headers.

Different forms of imageflow will also have different security policy defaults regarding from where it acceptable to fetch and to where it is acceptable to store resources (as you would expect). Clients or bindings which are backend-agnostic (re: lib/server/tool form) may have to buffer files to accommodate differences.

Bugs

I've been awake and working for 31 hours at the time of this writing, and at least 26 were spent coding. I'm quite sure you'll find many bugs - my commits do not seem to spare any hour of the day according to GitHub's super-cool Punch Card feature, which I presume was invented to gamify overtime. That said, I'm writing a tremendous amount of safe Rust code - recently passing number of lines of active C code in the library. Less than 3% of the Rust code base is marked as unsafe according to cargo count --unsafe-statistics, which means I can do more 4am coding without endangering my (eventual) night's sleep. Deleting 4kloc of the highest-risk C codebase on the 16th felt very, very good.

So, about reporting those bugs. You need to collect a few things:

1. Run imageflow_tool --version and lead with that
2. When imageflow_tool crashes, set RUST_BACKTRACE=1 or export RUST_BACKTRACE=1 depending on your OS, and re-run it. Now you should have a full stacktrace with (hopefully) resolved filenames and line numbers. This can't be a default setting, yet.
3. Include the exact command you're using to invoke Imageflow.
4. Make a how.txt file with the above sections of text.
5. Add the recipe.json file you're using
6. All input images
7. If it's not crashing, just producing the wrong result, add two things: a) what you would expect the result to be (as expected.txt or expected.png or expected.json as applicable), and b) the actual result.json and result.png/jpg files.
8. Put all this in a folder, ensure you can reproduce the problem self-contained in that folder, then zip it.
9. Open a GitHub issue and attach the zip file. Duplicate how.txt/expected.txt in the issue body.

I'll eventually replace the 6 copy & paste steps with 1 automated step, but you'll still have to open a Github issue and describe expectation differences yourself (unless it's a hard crash).

Retrospective on my velocity expectations vs. reality

• JSON API development has proceeded much, much faster than expected.
• Rust development productivity is (now) much better than projected.
• Compile times are unexpectedly long and have quadrupled this month (as has the quantity of Rust code, to be fair). We're seeing 25-32 minute AppVeyor and Travis builds per platform (Win64, Win32, OS X, Ubuntu.16.04, Ubuntu 14.04.
• Back-end work tasks on the graph engine and testing turned out to require more time, and took priority over creating the ImageResizer compatibility layer. My expectation for a beta server release (with an ImageResizer-compatible front-end) is inching into January right now. The clock doesn't start on licensing until there's an official beta release of this.
• Experimental codec improvements and automatic codec selection/tuning are looking to be 2018 work.
• Incorrect forum posts about certain aspects of Rust led to a few very painful curve-balls. About a month was lost in search of workarounds, but things are looking good.
• It's feasible that I could replace much more of Imageflow's C code than seemed likely. I've already replaced more than planned for v1, but development speed has been improving significantly as more up-front porting occurs. Rust is an extremely productive language once you've scaled the learning curve.
• I don't trust giflib, and pulled GIF support for now. I'm interested to know how many would prioritize GIF over 8-bit PNG support? It will take a few weeks to develop complete animated GIF support.
• The FFI API stabilized much faster than expected. We're not doing async in 1.0, so it's pretty simple and easy to write bindings for in any language. Basic Ruby bindings took 2 days. Interested? Docs are decent, and you can generate the low-level bindings from these headers. Please open an issue or e-mail [email protected] if you want to help with or start a new set of bindings; you'll be provided priority support and assistance.
• Building fluent, chain-able builders for generating graphs is actually quite easy - see what we do in fluent.rs. Using a DAG as the underlying representation of per-frame ( framewise in API parlance) operations has been working more smoothly than expected.

Call to action

• Run imageflow_tool examples --generate, and open issues
• Speak up if you're interested in working on bindings for your favourite language (I can help).
• Participate in relevant issue discussions.
• Share challenging images and tasks