Azure:Cloud Service stood up for every VM?

[timothyeharris]

I'm running PerfkitBenchmarker against Microsoft Azure, and I'm noticing the system creates a cloud service for every VM it stands up. This is not a realistic use of the cloud service. In Azure, Cloud Service is similar to a GCE "Project." In fact, I would suggest that the behavior against Azure be modified such that if the user specifies a --project parameter at runtime, that you use it just like you would for GCP - to put all the VMs for the benchmarks into a cloud service with that name. Otherwise, I would recommend that a cloud service only be stood up against the run-id, and not for every VM.

[voellm]

+Eric

This is a very good suggestion. I'll put this higher on the priority
list. We have a couple of tweaks to make in general to networking.

On Mon, Mar 2, 2015 at 6:09 PM, Tim Harris [email protected] wrote:

I'm running PerfkitBenchmarker against Microsoft Azure, and I'm noticing
the system creates a cloud service for every VM it stands up. This is
not a realistic use of the cloud service. In Azure, Cloud Service is
similar to a GCE "Project." In fact, I would suggest that the behavior
against Azure be modified such that if the user specifies a --project
parameter at runtime, that you use it just like you would for GCP - to put
all the VMs for the benchmarks into a cloud service with that name.
Otherwise, I would recommend that a cloud service only be stood up against
the run-id, and not for every VM.

Reply to this email directly or view it on GitHub
#139.

[timothyeharris]

The same concept holds true for affinity groups. You should really only have to stand one up and tear it down once, rather than doing it in between each benchmark. Creating an affinity group for the run-id and using it through all the benchmarks seems like that would be better than what's happening now.

[ehankland]

You can't easily add another VM to the same service (if you want to ssh into both) - the endpoints in the service have to be unique. If you try to create another VM with the the same ssh port open you get this error:

error: BadRequest : Port 22 is already in use by one of the endpoints in this deployment. Ensure that the port numbers are unique across endpoints within a deployment.
info: Error information has been recorded to azure.err
error: vm create command failed

We did initially try to do what you were suggesting, but we ran into this. If there's an easy way around this problem please let me know.

[ehankland]

As for the affinity groups, that's a different ask - it should definitely be possible to share pkb "networks" between benchmarks with the same run uri. It will take some amount of work though, because the way pkb is currently structured, benchmarks don't share anything.

[ehankland]

I should also probably note that, by default, the "azure vm create" command will automatically create a cloud service per VM for you, so I'd argue that this is fairly realistic compared to someone creating VMs from the linux CLI.

[ehankland]

Seems like changing the --ssh option works for remapping the ssh port - using one cloud service will definitely be possible in this case, but will also mean changing the firewall class as well. Just to be clear, the effect of doing this would be that all VMs would share the same "external" ip.

[timothyeharris]

The nomenclature within Azure is a bit different, I think. Creating a VM inside a Web Service is "adding a role". So using the --connect parameter to create the VM gives you the ability to connect it to an existing Web Service. The cli --help is not really all that helpful in this case :-(

Here's a stackoverflow article on the issue: http://stackoverflow.com/questions/23959017/cant-add-new-vm-to-existing-cloud-service-in-azure

[ehankland]

Yep - I tried using the -c option and that was what was causing the call to fail like I mentioned above until I changed the --ssh parameter to a port other than 22 - after that it worked.

[timothyeharris]

That makes sense since they're both on the same external IP address. For benchmarks dealing only with the "internal" resources of the cloud platform, storage, and network, I think this is a good approach (it's more like simulating a multi-tier application). If, however, benchmarks ever get added that require external connectivity, then it's better to have each stood up as a separate entity (that said, trying to "benchmark" when the external connection comes in to play is a bit of a non starter).

For all three (and future) platforms, it seems like PKB could have a "global prepare" that happens at the front end to stand up projects, affinity groups, networks, etc rather than standing all these up and tearing them all down in between each benchmark. Cleaning these up could happen in a "global cleanup" section once all the benchmarks have run.

I could see, in the future, wanting to stand up multiple "global containers" -- say in Azure AND GCE -- and run some of the multi-machine benchmarks with member-nodes in each :-)

[ehankland]

Moving to one shared service has the downside of slowing down vm/disk creation and deletion. If you try to do more than one operation at a time you get the error:

Windows Azure is currently performing an operation with x-ms-requestid ***** on this deployment that requires exclusive access.

This gets retried and eventually succeeds, but it does slow things down.
The cluster_boot benchmark with 5 vms took 545 seconds using the old approach and 831 seconds with this approach. Does having one shared service seem worth it given that down side? If so I can put the change out for review.

[timothyeharris]

I have pinged some engineering folks to get their guidance on what they see as the "right way" to do this.

[voellm]

@timothyeharris - Did you every get any follow-up on this bug? Still and issue?

[voellm]

@ehankland - What is the state of this?

[ehankland]

We still use one service per VM. I was waiting to hear back to see if there was a better way of doing things.

[timothyeharris]

I have not gotten any "preferred" advice from engineering (and due to organizational changes, am no longer working on this project). Sorry :(

Sent from Outlook Mobilehttps://aka.ms/qtex0l

On Thu, Dec 17, 2015 at 9:41 AM -0800, "ehankland" <[email protected]mailto:[email protected]> wrote:

We still use one service per VM. I was waiting to hear back to see if there was a better way of doing things.

Reply to this email directly or view it on GitHubhttps://github.com//issues/139#issuecomment-165524514.