-
Notifications
You must be signed in to change notification settings - Fork 6
OPP UX Pipeline and SLAs
The UX CD Pipeline allows you to integrate WebPageTests into your build process. You can also set SLAs on your builds as well.
I'm an expert... just cut to the chase...
- Create your WPT Script
- Create your groovy wrapper and checkin to whatever repo you want to use
- Run the builder job in jenkins
- Run your job with the Jenkins CLI
At this point, you may want to add SLAs to your script. Go back to step 2 and add your SLAs. Then repeat steps 3 and 4.
WebPageTest script is fairly easy. Use the example and link below to get you started.
WPT Scripting documentation: https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting
**Basic Script**
// turn off logging.
logData 0
// navigate to our login api
navigate https://ui.s1.constantcontact.com/roving/wdk/API_LoginSiteOwner.jsp?loginName=USERNAME&loginPassword=123456
// turn on logging - getting ready to record data for my page
logData 1
// increasing this activity wait value from the default 2 seconds to 15 seconds. This page can be slow and loads an iframe. WPT sometimes thinks its done when it's not.
setActivityTimeout 10000
// navigating to my page
navigate https://ui.s1.constantcontact.com/rnavmap/distui/reporting?hasReporting=true
// turning off the logging of data and logging out
logData 0
navigate https://ui.s1.constantcontact.com/rnavmap/evaluate.rnav?activepage=site.logout.eval
Creating Your Job To integrate a Web Page Test script with the Jenkins Performance UX Job Builder, the WPT script must be encapsulated inside a groovy script. The UX Job Builder reads all the WPT groovy files in the Tests directory of the Git repository: foo/WPT. The end result is a WPT jenkins job that will support SLAs and have trending that will be tracked and trended either going to your internal WPT instance http://wpt.mydomain.com or by using OPP UX.
For full instructions, use the link below: Web Page Test Jenkins Integration
The test label parameter for your job must be in the following format:
{env}.{appName}.{pageOrTestName}.{location}.{browser}.{connectionType}.{description-optional}
Please note we have standardized this syntax for a test label. Any jobs not using that test label format will be rejected.
Using the Jenkins CLI is a very simple way to call our jobs remotely. Download CLI: https://you-jenkins-server/jnlpJars/jenkins-cli.jar
java -jar jenkins-cli.jar -noKeyAuth -s https://your-jenkins-server/ build prod.myapp.mypage.us-east.firefox.cable -s -v -r 3
You can also override or pass in additional parameters using the "-p" option. In the example below we are setting the number of runs to 10 and only doing the first view of page (fresh cache) Jenkins CLI call to start a job and override parameters
java -jar jenkins-cli.jar -noKeyAuth -s https://your-jenkins-server/ build prod.myapp.mypage.us-east.firefox.cable -s -v -r 3 -p runs=10 -p fvonly=1
You may want to pass or fail your pipeline by the result of your UX performance jenkins job.
If you are following this page in order, you will have already completed the following:
- Created WPT script
- Created and committed your groovy script that includes your WPT script
- Setup your pipeline to call your UX job using the Jenkins CLI
The next step is to set SLAs. SLAs will allow your job to pass or fail. If your SLAs fail, your job will fail.
Performance Service Level Agreements (SLAs) allow us to pass or fail our tests based on performance based requirements.
Setting SLAs is as easy as copying an pasting json into your Jenkins job or adding it to your groovy script. Below is a simple example of how to set SLAs with JSON.
{
"successfulFVRuns": 6,
"median": {
"firstView": {
"TTFB": 300,
"SpeedIndex": 3000,
"visualComplete": 3000,
"userTime.galileo.loaded": 4000,
"userTime.ctct_fully_loaded": 7000,
"userTime.ctct_visually_complete": 3000,
"breakdown": {
"js": {
"requests": 5
},
"css": {
"requests": 5
}
}
}
}
}
In the above example, we are ensuring the following:
- We had 6 successful test runs
- TTFB is less than 300ms
- SpeedIndex is less than 3000ms
- Visual Complete is less than 3000ms
- Check 3 custom user timings
- We have 5 or less JS and CSS files
Note that in 2 thru 6 we are looking at median->firstView (median and uncached) page metrics. You can also replace median with average, min, max, and standardDeviation. You can also look at cached views as well by changing the firstView to repeatView.
You may also want to store your results in graphite. To do this just set the jenkins parameter "saveToGraphite" to TRUE Your test results will show up in your configured graphite cluster (https://your-graphite-server/) under: ux.{testLabel}
Where your test label is: {env}.{appName}.{pageOrTestName}.{location}.{browser}.{connectionType}.{description-optional} Please note we have standardized this syntax for a test label. Any jobs not using that test label format will be rejected.
The definition for most of these values can be found here: https://sites.google.com/a/webpagetest.org/docs/advanced-features/raw-test-results Below is a sample of all the options you can set
{
"successfulFVRuns": 6,
"successfulRVRuns": 6,
"min": "SAME SCHEMA AS MEDIAN",
"max": "SAME SCHEMA AS MEDIAN",
"median": {
"firstView": {
"loadTime": 2207,
"TTFB": 138,
"bytesOut": 69149,
"bytesOutDoc": 64087,
"bytesIn": 151288,
"bytesInDoc": 150491,
"connections": 1,
"requestsDoc": 30,
"responses_200": 25,
"responses_404": 0,
"responses_other": 2,
"result": 99999,
"render": 630,
"fullyLoaded": 2317,
"cached": 0,
"docTime": 2207,
"domTime": 0,
"score_cache": 40,
"score_cdn": 40,
"score_gzip": 100,
"score_cookies": -1,
"score_keep-alive": 100,
"score_minify": -1,
"score_combine": 100,
"score_compress": -1,
"score_etags": -1,
"gzip_total": 124521,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 1,
"aft": 0,
"domElements": 323,
"pageSpeedVersion": "1.9",
"title": "",
"titleTime": 0,
"loadEventStart": 2213,
"loadEventEnd": 2220,
"domContentLoadedEventStart": 441,
"domContentLoadedEventEnd": 487,
"lastVisualChange": 3099,
"browser_name": "Firefox",
"browser_version": "28.0.0.5186",
"server_count": 1,
"server_rtt": 2,
"base_page_cdn": "",
"fixed_viewport": 0,
"score_progressive_jpeg": -1,
"firstPaint": 0,
"docCPUms": 2043.613,
"fullyLoadedCPUms": 3213.621,
"docCPUpct": 92,
"fullyLoadedCPUpct": 19,
"isResponsive": -1,
"date": 1439610893,
"SpeedIndex": 3100,
"visualComplete": 0,
"effectiveBps": 69430,
"effectiveBpsDoc": 72736,
"run": 6,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
"domains": {
"moc.tcatnoctnatsnoc.1s.iu": {
"bytes": 14730,
"requests": 2,
"connections": 1
},
"moc.tcatnoctnatsnoc.1s.iutsid": {
"bytes": 3130,
"requests": 7,
"connections": 2
},
"ten.cdrtmo.tt.tcatnoctnatsnoc": {
"bytes": 8047,
"requests": 7,
"connections": 2
}
},
"breakdown": {
"html": {
"bytes": 114785,
"requests": 5
},
"js": {
"bytes": 10741,
"requests": 15
},
"css": {
"bytes": 0,
"requests": 0
},
"image": {
"bytes": 8872,
"requests": 3
},
"flash": {
"bytes": 0,
"requests": 0
},
"font": {
"bytes": 0,
"requests": 0
},
"other": {
"bytes": 8762,
"requests": 9
}
}
},
"repeatView": {
"loadTime": 2226,
"TTFB": 111,
"bytesOut": 83609,
"bytesOutDoc": 80843,
"bytesIn": 167476,
"bytesInDoc": 166844,
"connections": 3,
"requestsDoc": 30,
"responses_200": 26,
"responses_404": 0,
"responses_other": 2,
"result": 99999,
"render": 596,
"fullyLoaded": 2287,
"cached": 1,
"docTime": 2226,
"domTime": 0,
"score_cache": 40,
"score_cdn": 40,
"score_gzip": 100,
"score_cookies": -1,
"score_keep-alive": 100,
"score_minify": -1,
"score_combine": 100,
"score_compress": -1,
"score_etags": -1,
"gzip_total": 143186,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 1,
"aft": 0,
"domElements": 323,
"pageSpeedVersion": "1.9",
"title": "",
"titleTime": 0,
"loadEventStart": 2197,
"loadEventEnd": 2198,
"domContentLoadedEventStart": 383,
"domContentLoadedEventEnd": 424,
"lastVisualChange": 3078,
"browser_name": "Firefox",
"browser_version": "28.0.0.5186",
"server_count": 1,
"server_rtt": 1,
"base_page_cdn": "",
"fixed_viewport": 0,
"score_progressive_jpeg": -1,
"firstPaint": 0,
"docCPUms": 2152.814,
"fullyLoadedCPUms": 3447.622,
"docCPUpct": 96,
"fullyLoadedCPUpct": 20,
"isResponsive": -1,
"date": 1439610943,
"SpeedIndex": 3100,
"visualComplete": 0,
"effectiveBps": 76965,
"effectiveBpsDoc": 78886,
"run": 6,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
"domains": {
"moc.tcatnoctnatsnoc.1s.iu": {
"bytes": 14728,
"requests": 2,
"connections": 1
},
"moc.tcatnoctnatsnoc.1s.iutsid": {
"bytes": 19313,
"requests": 7,
"connections": 2
}
},
"breakdown": {
"html": {
"bytes": 114781,
"requests": 5
},
"js": {
"bytes": 27172,
"requests": 16
},
"css": {
"bytes": 0,
"requests": 0
},
"image": {
"bytes": 5692,
"requests": 3
},
"flash": {
"bytes": 0,
"requests": 0
},
"font": {
"bytes": 0,
"requests": 0
},
"other": {
"bytes": 8499,
"requests": 8
}
}
}
},
"average": {
"firstView": {
"loadTime": 2424,
"TTFB": 107.333336,
"bytesOut": 73925.836,
"bytesOutDoc": 64773.5,
"bytesIn": 148638.17,
"bytesInDoc": 147234.33,
"connections": 1.1666666,
"requestsDoc": 30,
"responses_200": 25.166666,
"responses_404": 0,
"responses_other": 2,
"result": 99999,
"render": 582.1667,
"fullyLoaded": 3867.5,
"cached": 0,
"docTime": 2424,
"domTime": 0,
"score_cache": 40,
"score_cdn": 40,
"score_gzip": 100,
"score_cookies": -1,
"score_keep-alive": 100,
"score_minify": -1,
"score_combine": 100,
"score_compress": -1,
"score_etags": -1,
"gzip_total": 127197,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 1,
"aft": 0,
"domElements": 323,
"pageSpeedVersion": 1.9,
"title": 0,
"titleTime": 0,
"loadEventStart": 2420.6667,
"loadEventEnd": 2425.8333,
"domContentLoadedEventStart": 455,
"domContentLoadedEventEnd": 491.5,
"lastVisualChange": 3242.6667,
"browser_name": 0,
"browser_version": 28,
"server_count": 1,
"server_rtt": 1.1666666,
"base_page_cdn": 0,
"fixed_viewport": 0,
"score_progressive_jpeg": -1,
"firstPaint": 0,
"docCPUms": 2139.8137,
"fullyLoadedCPUms": 3421.622,
"docCPUpct": 88.833336,
"fullyLoadedCPUpct": 18.333334,
"isResponsive": -1,
"date": 1.43961062E9,
"SpeedIndex": 3233.3333,
"visualComplete": 0,
"effectiveBps": 55116,
"effectiveBpsDoc": 64836.168,
"avgRun": 5,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
},
"repeatView": {
"URL": 0,
"loadTime": 2233.6667,
"TTFB": 118.333336,
"bytesOut": 84482.664,
"bytesOutDoc": 79675.836,
"bytesIn": 159151,
"bytesInDoc": 158264.67,
"connections": 2.1666667,
"requestsDoc": 28.833334,
"responses_200": 24.166666,
"responses_404": 0,
"responses_other": 2,
"result": 99999,
"render": 611.5,
"fullyLoaded": 2318.3333,
"cached": 1,
"docTime": 2233.6667,
"domTime": 0,
"score_cache": 17.5,
"score_cdn": 17.5,
"score_gzip": 100,
"score_cookies": -1,
"score_keep-alive": 100,
"score_minify": -1,
"score_combine": 100,
"score_compress": -1,
"score_etags": -1,
"gzip_total": 129474,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 1,
"aft": 0,
"domElements": 323,
"pageSpeedVersion": 1.9,
"title": 0,
"titleTime": 0,
"loadEventStart": 2227.6667,
"loadEventEnd": 2233,
"domContentLoadedEventStart": 400,
"domContentLoadedEventEnd": 437.5,
"lastVisualChange": 3099.3333,
"browser_name": 0,
"browser_version": 28,
"server_count": 1,
"server_rtt": 1,
"base_page_cdn": 0,
"fixed_viewport": 0,
"score_progressive_jpeg": -1,
"firstPaint": 0,
"docCPUms": 2165.814,
"fullyLoadedCPUms": 3473.6223,
"docCPUpct": 96.666664,
"fullyLoadedCPUpct": 20,
"isResponsive": -1,
"date": 1.43961062E9,
"SpeedIndex": 3100,
"visualComplete": 0,
"effectiveBps": 72371,
"effectiveBpsDoc": 74826.664,
"avgRun": 6,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
}
},
"standardDeviation": {
"firstView": {
"loadTime": 349,
"TTFB": 13,
"bytesOut": 9555,
"bytesOutDoc": 738,
"bytesIn": 2164,
"bytesInDoc": 2570,
"connections": 0,
"requests": 0,
"requestsDoc": 0,
"responses_200": 0,
"responses_404": 0,
"responses_other": 0,
"result": 0,
"render": 49,
"fullyLoaded": 3055,
"cached": 0,
"docTime": 349,
"domTime": 0,
"score_cache": 0,
"score_cdn": 0,
"score_gzip": 0,
"score_cookies": 0,
"score_keep-alive": 0,
"score_minify": 0,
"score_combine": 0,
"score_compress": 0,
"score_etags": 0,
"gzip_total": 1586,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 0,
"aft": 0,
"domElements": 0,
"pageSpeedVersion": 0,
"title": 0,
"titleTime": 0,
"loadEventStart": 349,
"loadEventEnd": 349,
"domContentLoadedEventStart": 72,
"domContentLoadedEventEnd": 73,
"lastVisualChange": 349,
"browser_name": 0,
"browser_version": 0,
"server_count": 0,
"server_rtt": 0,
"base_page_cdn": 0,
"fixed_viewport": 0,
"score_progressive_jpeg": 0,
"firstPaint": 0,
"docCPUms": 125,
"fullyLoadedCPUms": 255,
"docCPUpct": 6,
"fullyLoadedCPUpct": 1,
"isResponsive": 0,
"date": 174,
"SpeedIndex": 344,
"visualComplete": 0,
"effectiveBps": 19987,
"effectiveBpsDoc": 8605,
"avgRun": null,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
},
"repeatView": {
"URL": 0,
"loadTime": 34,
"TTFB": 5,
"bytesOut": 1768,
"bytesOutDoc": 1266,
"bytesIn": 6294,
"bytesInDoc": 6413,
"connections": 0,
"requests": 0,
"requestsDoc": 0,
"responses_200": 1,
"responses_404": 0,
"responses_other": 0,
"result": 0,
"render": 32,
"fullyLoaded": 40,
"cached": 0,
"docTime": 34,
"domTime": 0,
"score_cache": 18,
"score_cdn": 18,
"score_gzip": 0,
"score_cookies": 0,
"score_keep-alive": 0,
"score_minify": 0,
"score_combine": 0,
"score_compress": 0,
"score_etags": 0,
"gzip_total": 8222,
"gzip_savings": 0,
"minify_total": 0,
"minify_savings": 0,
"image_total": 0,
"image_savings": 0,
"optimization_checked": 0,
"aft": 0,
"domElements": 0,
"pageSpeedVersion": 0,
"title": 0,
"titleTime": 0,
"loadEventStart": 41,
"loadEventEnd": 41,
"domContentLoadedEventStart": 8,
"domContentLoadedEventEnd": 7,
"lastVisualChange": 29,
"browser_name": 0,
"browser_version": 0,
"server_count": 0,
"server_rtt": 0,
"base_page_cdn": 0,
"fixed_viewport": 0,
"score_progressive_jpeg": 0,
"firstPaint": 0,
"docCPUms": 14,
"fullyLoadedCPUms": 110,
"docCPUpct": 1,
"fullyLoadedCPUpct": 0,
"isResponsive": 0,
"date": 175,
"SpeedIndex": 0,
"visualComplete": 0,
"effectiveBps": 3327,
"effectiveBpsDoc": 3024,
"avgRun": null,
"userTime.galileo.editor-started": 87,
"userTime.galileo.editor-loading": 170,
"userTime.galileo.editor-starting": 95,
}
},
"statusCode": 200,
"statusText": "Test Complete"
}