Control for OS/Package/Compiler differences

[timothyeharris]

Would it make sense to create a "canonical" .VHD (and docker image) to include in with PerfKitBenchmarker that has all the appropriate libraries, compilers, and benchmark packages pre-installed?

Since it's possible to upload and use your own VHD/Container with all the cloud providers, this would make it possible to eliminate base OS differences as a source of problems with the benchmarks, and a source of argument for benchmark variability.

A pre-req for using perfkitbenchmarker could then be to upload the canonical image into your account/subscription for use by the tool.

(There's nothing really stopping individuals from doing this today, I suppose. Just wondering if a "blessed" version makes sense)

[voellm]

We are staying away from custom images because we want to see the perf of
the vendor provided images.

The hiccups in config are things we can communicate to Canonical to fix on
their end.
On Mar 6, 2015 12:53 AM, "Tim Harris" [email protected] wrote:

Would it make sense to create a "canonical" .VHD (and docker image) to
include in with PerfKitBenchmarker that has all the appropriate libraries,
compilers, and benchmark packages pre-installed?

Since it's possible to upload and use your own VHD/Container with all the
cloud providers, this would make it possible to eliminate base OS
differences as a source of problems with the benchmarks, and a source of
argument for benchmark variability.

A pre-req for using perfkitbenchmarker could then be to upload the
canonical image into your account/subscription for use by the tool.

(There's nothing really stopping individuals from doing this today, I
suppose. Just wondering if a "blessed" version makes sense)

—
Reply to this email directly or view it on GitHub
#150.

[timothyeharris]

Fair enough. The biggest hits I'm seeing to end-to-end runtime right now are coming from "getting the environment set up" -- because of packages not being installed correctly on the vendor supplied images. By "canonical" I didn't mean the vendor, rather canon in the religious sense :-)

[voellm]

Which in itself is interesting in a way.
On Mar 7, 2015 7:00 PM, "Tim Harris" [email protected] wrote:

Fair enough. The biggest hits I'm seeing to end-to-end runtime right now
are coming from "getting the environment set up" -- because of packages not
being installed correctly on the vendor supplied images.

—
Reply to this email directly or view it on GitHub
#150 (comment)
.

[voellm]

Before I close this out I should mention there is a --image option for using custom images. What we dont have is the ability to ignore package install.

Should we change this to a feature request to ignore installing packages to allow truly custom images?

[timothyeharris]

I am planning to use the --image option to handle this for own use case, yes :-)
I think there are two ways to go:

1. A flag to not install packages -- this would be fine for me.
2. Check to see if a package is already installed, and only attempt installation if it's not. -- This would also work for me, and would probably be a little more robust.

[voellm]

Let's go the check if already installed route. Thought we already did this.

[cmccoy]

We do verify that packages aren't installed multiple times during the same run, and package installs will be a noop if the system already has it. Anything compiled from source will always be installed, however.

[timothyeharris]

It might... (or it may just rely on the package installer to say "I already have this")

I'm looking for a way to make it easy to compare results between different instance types and know that what went into the end to end runtime is the same... i.e. I don't want to compare a result where I have the overhead of package installs on one instance and not on another....

For my use case, taking the --image route will solve this problem.