-
Notifications
You must be signed in to change notification settings - Fork 519
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
If we patch UnixBench - should we continue to call it UnixBench? #179
Comments
As it was requested during the meeting, here's some UnixBench data for different number of vCPUs. UnixBench System Benchmark Index Score data from Rackspace IAD region for same instance type family (same hardware) collected in October 2014. Performance2 instance type family does not over-allocates CPUs. vCPU:pCPU is 1:1. A pCPU (physical CPU) is a hardware thread in this case. Processor: Intel(R) Xeon(R) CPU E5-2670 Image: Ubuntu 14.04 LTS (PVHVM) These are the flavors (instance types) used:
This data was collected running with the same UnixBench patch > 16 CPUs used in PerfKit:
As more vCPUs are used with each instance type, beyond 16 vCPUs rate of improvement in the unixbench score goes down due to cross-socket memory latency, and cache effects. |
I think we should first attempt to contribute to the original benchmark. It that fails (or seems too slow) we should proceed with our own patch and a renaming scheme like what Ivan suggests. Have we tried to update the original Benchmark? |
Here's the original repo: We could try proposing the change, but I see two problems:
The patch used in PerfKit is the same patch that was proposed in the issue reported on their repo, which has been opened since 2011 Finally, I argue that the patch doesn't change the workload's behavior, and that it enables UnixBench to continue to scale up the parallel copies job just like it does so for CPUs < 16 for the following reasons. Here's why, you might want to get some coffee first :) By default in the code, UnixBench will run multiple iterations of two jobs. The first one is the single copy of the benchmarks, and the second one is parallel copies of the same benchmarks. Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1770 But there's a hardcoded cap of up to 16 parallel copies for the system benchmarks. Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#109 The number of copies to run are determined by the number of CPUs in the system: If there are N CPUs then during the parallel copies job execution, Unixbench will execute N copies of the benchmark. Code here: https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#1160 The problem is that this only true for up to 16 copies. If N > maxCopies = 16 then the parallel copies job execution is skipped here: The patch applied, which is the same that was proposed in the issue above, removes this cap. It allows UnixBench to continue to spawn the same number of parallel copies as your number of CPUs in the system with no cap, hence the parallel copies job continues to scale for CPUs > 16 in the same way it did for CPUs < 16. Thanks for reading. |
Several issues have been opened against the original repository on Google code in the last couple of weeks, but none of the owners have replied to any of them. I've exported the project under my personal account in GitHub, as others have done. I'm not committing to maintaining the UnixBench benchmark code, but I do plan to fix some of the issues that have been opened for years against UnixBench. Here's the repo: https://github.com/meteorfox/byte-unixbench Also here's a tagged commit to the last commit done on the original repo: |
Christos, if anything I think we should take the change and name the benchmark UnixBench-patch. We'd take it on the Google and Rackspace named sets immediately and put to vote adding it to the default sets instead of the original UnixBench next time we meet or whenever you think we have consensus. The Cloud Harmony folks, who had concerns with the change, can always keep the original benchmark on their named set. Comments? thank you, |
Hi Ivan and Christos, Thank you very much for providing that detailed explanation of the UnixBench patch! I did bring up a previous concern about the UnixBench patch in our most recent meeting, but the explanation dismisses those concerns. I'm fine with the patch as well as the name Unixbench-patch. Best, |
I believe we should patch it to run on 16+ cores, this is not what this issue is about. Also we should probably patch it on UnixBench's own depo, also not what this issue is about.
The question here is that, assuming we do patch it (or any other benchmark), shall we continue to call it UnixBench? Or should we call it UnixBench++ or similar?
My proposal is to name it UnixBench.[patch_name] or UnixBench.patch_revision.
The text was updated successfully, but these errors were encountered: