Dev Contrib: UT with AWS

Unit Tests execution requires an Openstack endpoint platform ; Tips with Amazon EC2.

Openstack platform creation

DevStack installation

On Amazon AWS create on some region:

1. A Security group and authorize port HTTP 80 (Openstack main endpoint) and 9696 (Openstack neutron network API)
2. A Ubuntu 16.04 ~t2.large instance with a >10Go volume, linked to this security group
3. (A Balise/Tag to be sure to stop this instance every day, because cost is ~80$/month otherwise ; like scheduler:ec2-startstop = none;2300;utc;all with ec2-scheduler cloud formation template)

Install Openstack DevStack on this instance, using pike version (the last with keystone identity v2) ; add -b stable/pike at end of git clone command)

Check if all is OK by retrieve a token through API using demo user (AWS URL & admin password should be updated):

curl --request POST --include --header "Content-Type: application/json" \
     --data '{"auth":{"identity":{"methods":["password"],"password":{"user":{"name":"demo","domain":{"name":"default"},"password":"secret"}}},"scope":{"project":{"name":"demo","domain":{"name":"default"}}}}}' \
     http://ec2-x.y.y.region.compute.amazonaws.com/identity/v3/auth/tokens

--> HTTP/1.1 201 Created

Public API access

By default, Openstack devstack is using eth0 ip for services URL. So this AWS private ip is in response of keystone-identity API ... and not accessible publicly. Neutron & Nova URLs service API should be updated with AWS public URL. The most simple way is todo that at startup.

Create the script sudo vi /etc/init.d/devstack-service-urls-update with:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          Openstack DevStack service URLs update
# Required-Start:    $remote_fs $syslog $devstack
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Update Openstack DevStack URLs service at boot time
# Description:       Cf upper.
### END INIT INFO

export AWS_URL=$(curl --silent http://169.254.169.254/latest/meta-data/public-hostname)
echo "Current AWS URL: $AWS_URL"
export OS_PASSWORD=$(cat /opt/stack/devstack/local.conf | grep "^ADMIN_PASSWORD=" | cut -d "=" -f 2)
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://localhost/identity/v3
export OS_DEFAULT_DOMAIN=default
export OS_USERNAME=admin
export OS_PROJECT_NAME=admin

echo "Waiting Openstack DevStack Keystone available ..."
until $(curl --output /dev/null --silent --head --fail $OS_AUTH_URL); do
    printf '.'
    sleep 5
done
echo ""

echo "Updating neutron URL"
NEUTRON_ID=`openstack endpoint list | grep neutron | cut -d "|" -f 2 | xargs`
openstack endpoint set --url "http://$AWS_URL:9696/" $NEUTRON_ID

echo "Updating nova URL"
NOVA_ID=`openstack endpoint list | grep nova | grep -v nova_legacy | cut -d "|" -f 2 | xargs`
openstack endpoint set --url "http://$AWS_URL/compute/v2.1" $NOVA_ID

echo "Updating cinder v1 URL"
CINDER_V1_ID=`openstack endpoint list | grep cinder | grep -v cinderv | cut -d "|" -f 2 | xargs`
openstack endpoint set --url "http://$AWS_URL/volume/v1/\$(project_id)s" $CINDER_V1_ID

echo "Updating cinder v2 URL"
CINDER_V2_ID=`openstack endpoint list | grep cinderv2 | cut -d "|" -f 2 | xargs`
openstack endpoint set --url "http://$AWS_URL/volume/v2/\$(project_id)s" $CINDER_V2_ID

echo "Updating cinder v3 URL"
CINDER_V3_ID=`openstack endpoint list | grep cinderv3 | cut -d "|" -f 2 | xargs`
openstack endpoint set --url "http://$AWS_URL/volume/v3/\$(project_id)s" $CINDER_V3_ID

exit 0

Add permissions and execution at startup:

sudo chmod 755 /etc/init.d/devstack-service-urls-update
sudo update-rc.d devstack-service-urls-update defaults

Execute the script, or reboot to verify.

Openstack platform configuration

Test datas

On Openstack dasboard (with user demo and in demo project):

1. Create a keypair (ex: demo) to use in tests files
2. Allocate some floating ip to the project (required for some unit test and good code coverage)
3. Create a volume demo (~1Go)

These tasks could be done by script on DevStack host:

#!/bin/sh

export OS_PASSWORD=$(cat /opt/stack/devstack/local.conf | grep "^ADMIN_PASSWORD=" | cut -d "=" -f 2)
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://localhost/identity/v3
export OS_DEFAULT_DOMAIN=default
export OS_USERNAME=demo
export OS_PROJECT_NAME=demo

export KEY_PAIR_NAME=demo
echo "KeyPair check/creation..."
openstack keypair list | grep $KEY_PAIR_NAME
if [ $? -eq 1 ]; then
  echo "Creating keypair $KEY_PAIR_NAME"
  openstack keypair create $KEY_PAIR_NAME
else
  echo "Keypair $KEY_PAIR_NAME exists, cancel creation"
fi

export FLOATING_IP_NUMBER=5
echo "Floating IP check/creation..."
while [ $(openstack floating ip list | grep -v "ID" | grep -v "\-\-\-\-\-" | wc -l) -lt $FLOATING_IP_NUMBER ]; do 
  openstack floating ip create public
done

export VOLUME_NAME=demo
echo "Volume check/creation..."
openstack volume list | grep $VOLUME_NAME
if [ $? -eq 1 ]; then
  echo "Creating volume $VOLUME_NAME"
  openstack volume create --size 1 $VOLUME_NAME
else
  echo "Volume $VOLUME_NAME exists, cancel creation"
fi

TeamCity Openstack plugin tests files content

In TeamCity Openstack plugin source code, create tests files in directory cloud-openstack-server/src/test/resources (AWS URL & admin password should be updated).

test.v3.properties:

test.url=https://ec2-x.y.y.region.compute.amazonaws.com/identity/v3
test.identity=default:demo:default:demo
test.password=secret
test.region=RegionOne

test.v3.yml

openstack-test-teamcity-plugin:
  image: cirros-0.3.5-x86_64-disk
  flavor: m1.tiny
  network: private
  security_group: default
  key_pair: demo
  auto_floating_ip: true
  volume: test-volume,/dev/vdc

test.v2.properties:

test.url=https://ec2-x.y.y.region.compute.amazonaws.com/identity/v2.0
test.identity=default:demo:default:demo
test.password=secret
test.region=RegionOne

test.v2.yml

openstack-test-teamcity-plugin:
  image: cirros-0.3.5-x86_64-disk
  flavor: m1.tiny
  network: private
  security_group: default
  key_pair: demo