Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If we patch UnixBench - should we continue to call it UnixBench? #179

Open
ivansmf opened this issue Mar 18, 2015 · 6 comments
Open

If we patch UnixBench - should we continue to call it UnixBench? #179

ivansmf opened this issue Mar 18, 2015 · 6 comments
Assignees
Labels

Comments

@ivansmf
Copy link

ivansmf commented Mar 18, 2015

I believe we should patch it to run on 16+ cores, this is not what this issue is about. Also we should probably patch it on UnixBench's own depo, also not what this issue is about.

The question here is that, assuming we do patch it (or any other benchmark), shall we continue to call it UnixBench? Or should we call it UnixBench++ or similar?

My proposal is to name it UnixBench.[patch_name] or UnixBench.patch_revision.

@meteorfox
Copy link
Contributor

As it was requested during the meeting, here's some UnixBench data for different number of vCPUs.

UnixBench System Benchmark Index Score data from Rackspace IAD region for same instance type family (same hardware) collected in October 2014. Performance2 instance type family does not over-allocates CPUs. vCPU:pCPU is 1:1. A pCPU (physical CPU) is a hardware thread in this case.

Processor: Intel(R) Xeon(R) CPU E5-2670
20MB LLC
8 physical cores (16 threads) per socket
Turbo Mode On
2 Sockets, NUMA off for all flavors, except performance2-120 (which still is interleaved)

Image: Ubuntu 14.04 LTS (PVHVM)
Linux Kernel version: 3.13.0-36-generic (buildd@toyol)
GCC version: 4.8.2 (Ubuntu 4.8.2-19ubuntu1)
Hypervisor: Xen 4.1

These are the flavors (instance types) used:

  • performance2-15 (4 vCPUs)
  • performance2-30 (8 vCPUs)
  • performance2-60 (16 vCPUs)
  • performance2-90 (24 vCPUs)
  • performance2-120 (32 vCPUs)

This data was collected running with the same UnixBench patch > 16 CPUs used in PerfKit:

flavor vcpus score
performance2-15 4 3057.5
performance2-15 4 3175.4
performance2-30 8 4535.4
performance2-30 8 4424.7
performance2-60 16 6175.2
performance2-60 16 6507.6
performance2-90 24 7037.9
performance2-90 24 7108.3
performance2-120 32 7461
performance2-120 32 7394.8

As more vCPUs are used with each instance type, beyond 16 vCPUs rate of improvement in the unixbench score goes down due to cross-socket memory latency, and cache effects.

@kozyraki
Copy link

I think we should first attempt to contribute to the original benchmark. It that fails (or seems too slow) we should proceed with our own patch and a renaming scheme like what Ivan suggests.

Have we tried to update the original Benchmark?

@meteorfox
Copy link
Contributor

Here's the original repo:
https://code.google.com/p/byte-unixbench/

We could try proposing the change, but I see two problems:

  • The repo seems to be inactive because the last change to that repo was in Jan 17, 2011
  • The patch has already been proposed in an issue opened in March 6, 2011, and it hasn't been merged.

The patch used in PerfKit is the same patch that was proposed in the issue reported on their repo, which has been opened since 2011
Issue: https://code.google.com/p/byte-unixbench/issues/detail?id=4

Finally, I argue that the patch doesn't change the workload's behavior, and that it enables UnixBench to continue to scale up the parallel copies job just like it does so for CPUs < 16 for the following reasons.

Here's why, you might want to get some coffee first :)

By default in the code, UnixBench will run multiple iterations of two jobs. The first one is the single copy of the benchmarks, and the second one is parallel copies of the same benchmarks.

Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1770
Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1160

But there's a hardcoded cap of up to 16 parallel copies for the system benchmarks.

Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#109

The number of copies to run are determined by the number of CPUs in the system:
Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1775

If there are N CPUs then during the parallel copies job execution, Unixbench will execute N copies of the benchmark.

Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1160

The problem is that this only true for up to 16 copies. If N > maxCopies = 16 then the parallel copies job execution is skipped here:
https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1329

The patch applied, which is the same that was proposed in the issue above, removes this cap. It allows UnixBench to continue to spawn the same number of parallel copies as your number of CPUs in the system with no cap, hence the parallel copies job continues to scale for CPUs > 16 in the same way it did for CPUs < 16.

Thanks for reading.

@cmccoy cmccoy changed the title If we patch UnixBech - should we continue to call it UnixBench? If we patch UnixBench - should we continue to call it UnixBench? Mar 20, 2015
@meteorfox
Copy link
Contributor

Several issues have been opened against the original repository on Google code in the last couple of weeks, but none of the owners have replied to any of them.

I've exported the project under my personal account in GitHub, as others have done.

I'm not committing to maintaining the UnixBench benchmark code, but I do plan to fix some of the issues that have been opened for years against UnixBench.

Here's the repo: https://github.com/meteorfox/byte-unixbench

Also here's a tagged commit to the last commit done on the original repo:
https://github.com/meteorfox/byte-unixbench/tree/v5.1.3

@ivansmf
Copy link
Author

ivansmf commented Apr 3, 2015

Christos, if anything I think we should take the change and name the benchmark UnixBench-patch. We'd take it on the Google and Rackspace named sets immediately and put to vote adding it to the default sets instead of the original UnixBench next time we meet or whenever you think we have consensus.

The Cloud Harmony folks, who had concerns with the change, can always keep the original benchmark on their named set.

Comments?

thank you,
Ivan

@kennymuli
Copy link

Hi Ivan and Christos,

Thank you very much for providing that detailed explanation of the UnixBench patch!

I did bring up a previous concern about the UnixBench patch in our most recent meeting, but the explanation dismisses those concerns. I'm fine with the patch as well as the name Unixbench-patch.

Best,
Kenny Li
(Cloud Spectator)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants