Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use same date for all images #58

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nbyavuz
Copy link
Collaborator

@nbyavuz nbyavuz commented Mar 27, 2023

There is a CIRRUS_CHANGE_TIMESTAMP variable which is timestamp of when the commit is pushed. Convert that timestamp to date and use it at the images. So, all of the images' date part will be the same. Then we can use and update these images at the other CI tasks.

A downside comes to my mind, if the image is created and we re-run CI; it will fail because the image name will be same.

New method is:

Export date to artifact on one task which other tasks are depended. Then download this artifact on other tasks and use it to set date.

@nbyavuz nbyavuz requested a review from anarazel March 27, 2023 15:24
@anarazel
Copy link
Owner

I don't think that works - the images are rebuilt daily, and that will all use the same commit, no?

@nbyavuz
Copy link
Collaborator Author

nbyavuz commented Mar 27, 2023

Actually I thought the same but tested and found that CIRRUS_CHANGE_TIMESTAMP is different for cron jobs. For example:

These four tasks have same commit hash: d1ef15b
but have different timestamps. All timestamps show the timestamp of the when cron job started.

https://cirrus-ci.com/task/6486504743108608 -> CIRRUS_CHANGE_TIMESTAMP=1679607024158
https://cirrus-ci.com/task/4801905692508160 -> CIRRUS_CHANGE_TIMESTAMP=1679520624502
https://cirrus-ci.com/task/5849622967484416 -> CIRRUS_CHANGE_TIMESTAMP=1679434223266
https://cirrus-ci.com/task/5030596091576320 -> CIRRUS_CHANGE_TIMESTAMP=1679347823902

@anarazel
Copy link
Owner

Oh, interesting.

.cirrus.yml Outdated
@@ -17,6 +17,15 @@ env:
# GCP_REGION
# GCP_REPO

# Get the same date for the tasks. CIRRUS_CHANGE_TIMESTAMP is
# timestamp of when the commit is pushed in ms. Add 500ms to
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should note that the timestamp does change when task is triggered by cron.

.cirrus.yml Outdated
# timestamp of when the commit is pushed in ms. Add 500ms to
# round up to the nearest second then convert it to date. Another method
# is used at the Windows docker-builder task because $CIRRUS_ENV can't
# be updated and this method is not working on PowerShell.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't CIRRUS_ENV be updated? IIRC it was broken at some point, but after I reported that, cirrus implemented a fix?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't update them on Windows, it says:
'Failed collect CIRRUS_ENV subsystem results: CIRRUS_ENV file should contain lines in KEY=VALUE format'
but it is already in KEY=VALUE format.

Example task

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that's an encoding issue. Powershell will write utf-16 by default.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping ^

@anarazel
Copy link
Owner

Hm. I wonder if it's going to be confusing that an image with newer packages could have an older timestamp?

@anarazel
Copy link
Owner

What about this alternative: We have a first task that does nothing but export the date into a text artifcact, that the other tasks then use to determine the timestamp?

@nbyavuz
Copy link
Collaborator Author

nbyavuz commented Mar 27, 2023

What about this alternative: We have a first task that does nothing but export the date into a text artifcact, that the other tasks then use to determine the timestamp?

I didn't understand this, doesn't that text file stay on first task's machine, how will the other tasks reach it?

@nbyavuz
Copy link
Collaborator Author

nbyavuz commented Mar 27, 2023

Hm. I wonder if it's going to be confusing that an image with newer packages could have an older timestamp?

This should happen rarely, right? Otherwise, yes. It can be confusing.

@anarazel
Copy link
Owner

I didn't understand this, doesn't that text file stay on first task's machine, how will the other tasks reach it?

https://cirrus-ci.org/guide/writing-tasks/#current-build-artifacts

Even though that documents .zip, you can download individual artifacts directly. E.g. for
https://cirrus-ci.com/build/5278195855917056 I can download the output of the failed openbsd task with:
curl 'https://api.cirrus-ci.com/v1/artifact/build/5278195855917056/OpenBSD%20-%207%20-%20Meson/testrun/build/testrun/adminpack/regress/log/initdb.log'

Note that that does not include the task id of the openbsd build, just the build id shared by all the tasks.

For this purpose it'd be good to use a task name that doesn't include spaces, or has an alias without spaces.

@nbyavuz nbyavuz force-pushed the image-creation-features branch from 6d2c7d9 to 0e7c02b Compare March 28, 2023 12:35
@nbyavuz nbyavuz requested a review from anarazel March 28, 2023 14:39
Copy link
Owner

@anarazel anarazel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if a simpler solution could be implemented with starlark:
https://cirrus-ci.org/guide/programming-tasks/

We could make the starlark script emit something like

env:
   IMAGE_DATE: iso-date-string

which I think should suffice? We already do that for some vars, so it should be easy:
https://github.com/anarazel/pg-vm-images/blob/main/.cirrus.star

Apparently this should give us the time:
https://github.com/qri-io/starlib/blob/master/time/doc.go#LL27C1-L27C1

.cirrus.yml Outdated
# timestamp of when the commit is pushed in ms. Add 500ms to
# round up to the nearest second then convert it to date. Another method
# is used at the Windows docker-builder task because $CIRRUS_ENV can't
# be updated and this method is not working on PowerShell.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping ^

@nbyavuz nbyavuz force-pushed the image-creation-features branch from 0e7c02b to f2f6a7e Compare June 7, 2023 15:02
@nbyavuz
Copy link
Collaborator Author

nbyavuz commented Jun 7, 2023

Apparently this should give us the time: https://github.com/qri-io/starlib/blob/master/time/doc.go#LL27C1-L27C1

It seems time is not available in Cirrus Starlark.

@nbyavuz nbyavuz force-pushed the image-creation-features branch 2 times, most recently from 99ddf75 to a27f448 Compare June 8, 2023 06:55
Export date to artifact on one task which other tasks are depended.
Then download this artifact on other tasks and use it to set date.
@nbyavuz nbyavuz force-pushed the image-creation-features branch from a27f448 to 81108f3 Compare June 8, 2023 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants