Bug fixes

Please take a moment to try out imageflow_tool (JSON API) and imageflow_server (querystring API) and provide feedback.

We've only had 3 people provide feedback to date, yet I will soon have to start locking down aspects of the API design.

HTTPS resource fetching is alpha, and may not work everywhere. Bug reports still appreciated.

Docker support and more AVX

In addition to the baseline x86_64 SIMD support, we've added a couple extra builds, one with AVX and another with even more AVX thrown in. Like AVX_2_.

If you have a Skylake, Broadwell, Haswell, Ivy Bridge, or Sandy Bridge (2011+) CPU, try the -sandybridge- build for Linux. It requires AVX support, which the aforementioned families have (if you avoid Celeron-branded stuff).
If you have Haswell, Broadwell, or Skylake, try the -haswell- build, which requires AVX2 & friends.

Excluding perhaps t2.micro, all AWS, Azure, and Digital Ocean virtual machines should be Sandy Bridge or higher (most are actually Haswell or higher).

We're using the sandybridge build for our Docker Cloud demo. Most Digital Ocean servers support Haswell, but not all. #92 (comment)

imageflow_server start --demo

imageflow_server start --demo or imageflow_server --help

The demo page is just the 2009-era ImageResizer examples page; you'll probably want to move to start --mount pretty quickly.

You can specify any number of --mount /prefix/:provider:parameter arguments, where parameter is usually a directory or base URL to which incoming paths are concatenated.

Examples

• --mount /img/:ir4_local:./images/
• --mount /js/:static:./javascript/
• --mount /proxy_microsoft/:ir4_http:http:://microsoft.com/ Note the http:: - we're using : as a delimiter, so you must use :: to make it literal.
• --mount /staticfile_proxy/:permacache_proxy:http:://bucket.s3.amazonaws.com/
• --mount /githubproxy/:permacache_proxy_guess_content_types:http:://raw.githubusercontent.com/

imageflow_server v0.0.10 only supports the querystring API (ImageResizer4 compatible variant). It does not yet expose the JSON API. It can fetch originals over HTTP or from the local filesystem.

For now, please use the JSON API via imageflow_tool, which is better equipped for bug reporting. Imageflow_tool offers --debug-package [name], which automatically creates a zipfile I can use to reproduce any bugs you may encounter.

This is our first technical preview of imageflow_server. It is intentionally restricted to localhost connections.

What's missing

libimageflow, imageflow_tool, and imageflow_server all currently lack

• Operation cost prediction
• GIF and 8-bit PNG support
• Background color blending (needed for transparent PNG -> Jpeg conversion)
• Custom encoder enhancements.
• Windows optimizations (it's still about 3x slower than linux builds). On my Ubuntu box, demo page images are processing in 5-45ms (browser reports 20ms+ RT). Look for the X-Imageflow-Perf HTTP header in responses.
• Graph optimization (all builds do unnecessary work right now).
• ir4 providers unnecessarily re-encode all images, even when there's no querystring.
• Security review. Everything is still in technical preview for API design review and feedback.

What should work

• Jpeg, Png32, Png24, all with input color profile support.
• Jpeg EXIF rotation support.
• Jpeg entropy coding (smaller file sizes, slower encoding times). You can't turn this off yet, and and this is making encoding large jpegs a bit slow.
• Over 50% of planned JSON API operations: canvas, crop, rotate, flip, transpose, scale, sharpen, expand canvas, draw rectangle, decode, encode, copy,
  • Most ImageResizer 4 command keys: w, width, maxwidth, h, height, maxheight, mode, scale, flip, sflip, rotate, srotate, quality, format, cropxunits, cropyunits, zoom, bgcolor (for padding only), f.sharpen, anchor.
  • Fetching source images from disk, URL, or (within JSON) base64.
  • Permanent source caching for HTTP requests (no output caching)

Windows builds:
32-bit: https://ci.appveyor.com/api/buildjobs/yggrrbgyjc0kqux2/artifacts/imageflow-v0.0.10-37a156b-win-x86.zip
64-bit: https://ci.appveyor.com/api/buildjobs/6oe4gg06bx6m94rk/artifacts/imageflow-v0.0.10-37a156b-win-x64.zip

Introducing imageflow_tool

Introducing imageflow_tool

TLDR; run imageflow_tool examples --generate and start opening issues on GitHub about what's broke or you think we should change.

One of Imageflow's key design goals is a compete and strict JSON interface which is unified and consistent across the library, server, and command-line form. I still haven't written all the error responses, but I think this technical preview of imageflow_tool will give you a good look at the format I'm aiming for with the universal JSON API. This release doesn't include an RIAPI or ImageResizer compatibility layer.

Ways of accessing the universal JSON API today

• With imageflow_tool via imageflow_tool v0.1/build --json request.json
• With libimageflow via imageflow_job_send_json(struct imageflow_context* context, struct imageflow_job* job, char const* method, uint8_t const* json_buffer, size_t json_buffer_size)
• With libimageflow-rb via Imageflow::Context.send_json(method:, data:) (this Ruby gem is not yet well factored or documented for reuse, but is also quite simple)

And later

• With imageflow_server via http://host/imageflow/v0.1/build
• With community-provided language bindings (official support is for Ruby and eventually Node, but @SamuelEnglard is working on C# bindings.

We don't have a Rust server ready yet (that requires far more security testing and coverage), but imageflow_server will implement essentially the same JSON API. Minor differences for request signatures and authorization/authentication may be required, but can probably be handled via HTTP headers.

Different forms of imageflow will also have different security policy defaults regarding from where it acceptable to fetch and to where it is acceptable to store resources (as you would expect). Clients or bindings which are backend-agnostic (re: lib/server/tool form) may have to buffer files to accommodate differences.

Bugs

I've been awake and working for 31 hours at the time of this writing, and at least 26 were spent coding. I'm quite sure you'll find many bugs - my commits do not seem to spare any hour of the day according to GitHub's super-cool Punch Card feature, which I presume was invented to gamify overtime. That said, I'm writing a tremendous amount of safe Rust code - recently passing number of lines of active C code in the library. Less than 3% of the Rust code base is marked as unsafe according to cargo count --unsafe-statistics, which means I can do more 4am coding without endangering my (eventual) night's sleep. Deleting 4kloc of the highest-risk C codebase on the 16th felt very, very good.

So, about reporting those bugs. You need to collect a few things:

1. Run imageflow_tool --version and lead with that
2. When imageflow_tool crashes, set RUST_BACKTRACE=1 or export RUST_BACKTRACE=1 depending on your OS, and re-run it. Now you should have a full stacktrace with (hopefully) resolved filenames and line numbers. This can't be a default setting, yet.
3. Include the exact command you're using to invoke Imageflow.
4. Make a how.txt file with the above sections of text.
5. Add the recipe.json file you're using
6. All input images
7. If it's not crashing, just producing the wrong result, add two things: a) what you would expect the result to be (as expected.txt or expected.png or expected.json as applicable), and b) the actual result.json and result.png/jpg files.
8. Put all this in a folder, ensure you can reproduce the problem self-contained in that folder, then zip it.
9. Open a GitHub issue and attach the zip file. Duplicate how.txt/expected.txt in the issue body.

I'll eventually replace the 6 copy & paste steps with 1 automated step, but you'll still have to open a Github issue and describe expectation differences yourself (unless it's a hard crash).

Retrospective on my velocity expectations vs. reality

• JSON API development has proceeded much, much faster than expected.
• Rust development productivity is (now) much better than projected.
• Compile times are unexpectedly long and have quadrupled this month (as has the quantity of Rust code, to be fair). We're seeing 25-32 minute AppVeyor and Travis builds per platform (Win64, Win32, OS X, Ubuntu.16.04, Ubuntu 14.04.
• Back-end work tasks on the graph engine and testing turned out to require more time, and took priority over creating the ImageResizer compatibility layer. My expectation for a beta server release (with an ImageResizer-compatible front-end) is inching into January right now. The clock doesn't start on licensing until there's an official beta release of this.
• Experimental codec improvements and automatic codec selection/tuning are looking to be 2018 work.
• Incorrect forum posts about certain aspects of Rust led to a few very painful curve-balls. About a month was lost in search of workarounds, but things are looking good.
• It's feasible that I could replace much more of Imageflow's C code than seemed likely. I've already replaced more than planned for v1, but development speed has been improving significantly as more up-front porting occurs. Rust is an extremely productive language once you've scaled the learning curve.
• I don't trust giflib, and pulled GIF support for now. I'm interested to know how many would prioritize GIF over 8-bit PNG support? It will take a few weeks to develop complete animated GIF support.
• The FFI API stabilized much faster than expected. We're not doing async in 1.0, so it's pretty simple and easy to write bindings for in any language. Basic Ruby bindings took 2 days. Interested? Docs are decent, and you can generate the low-level bindings from these headers. Please open an issue or e-mail [email protected] if you want to help with or start a new set of bindings; you'll be provided priority support and assistance.
• Building fluent, chain-able builders for generating graphs is actually quite easy - see what we do in fluent.rs. Using a DAG as the underlying representation of per-frame ( framewise in API parlance) operations has been working more smoothly than expected.

Call to action

• Run imageflow_tool examples --generate, and open issues
• Speak up if you're interested in working on bindings for your favourite language (I can help).
• Participate in relevant issue discussions.
• Share challenging images and tasks