Skip to content

Latest commit

 

History

History
170 lines (130 loc) · 13.1 KB

README.md

File metadata and controls

170 lines (130 loc) · 13.1 KB
Raw

Gardener Website Generator

This repository contains the build tools, coniguration, web framework for the website and integration with CI/CD. For site source content including documentation, see the /gardener/documentation repo. The website home is /gardener/website/docs.

CI/CD Overview

The repositories involved in the CI/CD are:

  • /gardener/documentation is the Website Source Content repository. It contains the source content for the website, used by the builder to produce the static HTML to be served. This is the primary repository for contributions.
  • /gardener/website-generator(this repository) is the Website Generator repository. It contains the tools, the scripts and build configuration for the website, including all common framework html, styles, javascript and images, as well as the scripts and build configuration for the build environment container image.
  • /gardener/website is the Website home repository. It hosts the produced website content and is configured to serve it using GitHub Pages.

The website builds and deployments are orchestrated by Concourse CI/CD pipeline and triggered regularly (every 24h) or upon changes in /gardener/documentation or /gardener/website-generator repos. The build results are then pushed to /gardener/website/docs and served as GitHub Pages site.

Build

The build and deployment triggered by Concourse goes through the stages described here. Except the first two stages, the rest are all orchestrated by the ./ci/build script.

Run the website build container

  • Container Image: eu.gcr.io/gardener-project/gardener-website-generator
  • Environment: see the Dockerfile for details. Noteworthy mentions are:
    • NodeJS: used for pre-build operations preparing the site content for Hugo. The NodeJS used in the image is based on the following base image: eu.gcr.io/gardener-project/cpet/node-image:1.0.0
    • Hugo: the static site builder we use to produce the site.

Clone and setup repositories clones

  1. Clone the three repositories locally:
  2. Symlink the website folder from the documentation clone as content in the hugo folder in website-generator clone
    website-generator/hugo/content → documentation/website

Run the site pre-build operations

The stage is performed by a NodeJS script on the website content from website-generator/hugo/content (see the symlink step above) and goes through the following phases:

  1. Fetch remote pages configured in the website with front matter property remote: URL from other GitHub repositories.
  2. Fetch the git commits for all pages. This is later rendered by page components showing contributors and last page update.
  3. Parse all files and check if any link references a remote site, which has been imported and rewrite those links to reference the local (imported) page.

Build the site

The stage is performed by Hugo and goes through the following general phases:

  • transform the website content (Markdown) from website-generator/hugo/content (see the symlink step above) into static HTML
  • process shortcodes in the content, if any.
  • apply the appropriate layouts for the content type and configuration

The result of the site build is output to the /docs folder of the gardener repo clone.

Publish changes

The changes to the website home repo clone from previous phase are staged, commited and finally pushed to the gardener repo, where they are immediately served by GitHub Pages.

Build Locally

In most cases when developing site content, you will not need to run the full site build and al you need is a site preview. For the rest of the cases there are several options outlined below.

Feel free to reuse the image utilized by the CI/CD and avoid setting up the tools for the build environment. That has the advantage of keeping you up-to-date with the build setup effortlessly too. Clone the gardener-genrator repo in the contianer and use the website-generator/scripts/setup script to setup the other clones as required. After you have cloned the repos in the container, you can reuse the website-generator/.ci/build script for full scale builds. Currently, there is no end-to-end script for the setup stage, so you have to come up with something yourself mount and run it. Contributions are welcome.

In other rare case or when you cannot use Docker for some reason, see the procedure below how to setup build environment and run local build.

Prerequisites:

Procedure:

# Fork (if you plan ot make changes here) and clone this repository locally
$ git clone https://github.com/gardener/website-generator.git

# Change to the cloned repo 
cd website-generator

# Run make (or make setup) to have all necessary repos cloned, and setup (linked) for you automatically as necessary (if they do not exist).
$ make

# It is highly recommended to supply access token before starting the build to avoid Github API rate limit restrictions.
# For basic authentication (deprecated by GitHub), use the `GIT_USER` and `GIT_PASSWORD` instead.   
$ export GIT_OAUTH_TOKEN=<your-github-personal-access-token>

# Set AUTO_PUBLISH to false to instruct the build not to publish changes to the documentation and website repos. This is in the role of CI/CD builds
# and should not be taken over. 
$ export AUTO_PUBLISH=false

$ make build

Normally, all steps except make build are executed once during the initial setup. After that, the build step can then be executed numerous times.
The build results are produced in website/docs.

Build Configuration

The build is parameterized by means of environment variables.

Locations

The build will apply some heuristics and infer documentation and gardener are the names of clonded repos that are peer to website-generator. To override their paths, use the coresponding environment variables:

  • GARDENER_DOCUMENTATION_PATH sets the path to the documentation repo (default: /documentation)
  • GARDENER_WEBSITE_PATH sets the path to the build output repo (default: /website)
  • GARDENER_GENERATOR_PATH changes the infered location of website-generatordefaulting to the directory where the build script is executed.
  • CONTENT The build scripts will assume the source for building the site is available in hugo/content (note the symlink content<==> documentation/website we setup earlier). In case you need to override its location, use the CONTENT environment variable.

GitHub authentication

It is recommended to supply authentication credentials before starting the build to avoid Github API rate limit restrictions.
For token-based authentication (recommended), use the GIT_OAUTH_TOKEN environment variable. For basic authentication (deprecated by GitHub), use the GIT_USER and GIT_PASSWORD.

Build results

  • AUTO_PUBLISH controls the post-build steps publishing the results. Normally it should not be set, unless for the CI/CD pipeline definition build step. To enable, set to true.

Run Site Previews Locally

Site previews aid the process of developing site material or layouts. Similiar to builds, it is recommended to run site previews with docker, using the scripts supplied in /documentation for that. In case you can't benefit from that for some reason, the instructions for setting it up manually are listed here.

Prerequisites:

  • Cloned /documentation and /website-generator repos
  • Symlink /website-generator/hugo/content <==> /documentation/website
  • Hugo available on your system

The first two prerequisites are automatically ensured by running make (or make setup). Or you can do that manually.

Note that the /website repo is not necessary to perform this task.

Procedure:

cd website-generator/hugo

$ hugo serve

You can now explore the site and your changes upon save at http://localhost:1313.

Windows 10 Users

The instructions above are applicable when using Windows 10 WSL e.g. with Ubuntu. Here are a few hints how to setup your environment accordingly.

Symlinks in Windows

One of the problematic integration areas in WSL are symbolic links. Despite that a good progress was made it is still necessary to use Windows tools to create symbolic links on a Windows file system. The build and setup scripts above consider WSL environments and will automatically fallback to Windows mklink command instead of ln if necessary. A prerequisite to the success of this operation is to enable Windows Developer Mode (Settings > Update & Security > For developers > Use developer features).

Should you experience problems on this stage, create the symlink manually. The build script will work with existing content symlink and not attempt to create one. To create the content symlink in website-generator repo clone's hugo folder, start CMD as Administrator and make symlink to folder (/D)

[website-generator-repo-clone-path]>\hugo>mklink /D content <absolute-path-to-gardener-documentation-website-folder>

A successfull operation will signal output similiar to this:

symbolic link created for content <<===>> <actual-path-on-your-system>\website

Scripts line endings

Note that if you are using GitBash it is likely to be configured to change line endings with Windows (CRLF) instead of UNIX (LF) style. You will need to change line endings in the bash scripts you plan to use. If you use VS Code look in the lower right corner and you will notice CRLF, which can be changed to LF. Notepad++ also has a Line Ending change option. Consult with your favorite tool options how to deal with this best.

Docker

Install Docker Desktop for Windows if you plan to make use of the build image to preview changes or run the same full-scale build as the CI/CD.
A helpful article on setting up WSL to work flawlesly with Desktop Docker for Windows is available here. The list below is a TL;DR; for some key points.

  • Volumes mounting If you plan to use the image you will need to mount at least source content volume and probably a scripts volume. Docker and volume mounting has some subtleties when it comes to Windows. In short, make sure you have the C: drive shared in Docker Desktop and no firewall rule getting in the way, and ensure that host mount paths start with /c and not /mnt/c. If you set WSL=1 before running the scripts, they will try to remove the leading /mnt.
  • Enable using the daemon from Docker Desktop for Windows in WSL.
    1. Go to Docker Desktop > Settings and check the "Expose daemon on tcp://localhost:2375 without TLS" checkbox enabled.
    2. Setup .bashrc in WSL with the following environment variable 
      export DOCKER_HOST=tcp://localhost:2375

Troubleshooting

Concourse Builds

To troubleshoot failed website production pipeline in Concourse, use fly:

  1. Download fly (look for cli: at the bottom right of the screen at https://concourse.ci.gardener.cloud/)
  2. Login to a target with fly: 
    $ fly -t gardener login -c https://concourse.ci.gardener.cloud/ -n gardener
    Fly will request that you login at a URL and automatically intercept a successfull login.
  3. Intercept the failed container with 
    $ fly -t gardener hijack -u <url-of-your-build>
    Example is https://concourse.ci.gardener.cloud/teams/gardener/pipelines/gardener-website-generator-master/jobs/master-head-update-job/builds/102 , where 102 is the number of the build job that you wnat to inspect.