Skip to content

Latest commit

 

History

History
541 lines (402 loc) · 17.1 KB

README.md

File metadata and controls

541 lines (402 loc) · 17.1 KB
Raw

Ursus

Ursus is the static site generator used by All About Berlin and my personal website. It turns Markdown files and Jinja templates into a static website.

It also renders images in different sizes, renders SCSS, minifies JS and generates Lunr.js search indexes.

This project is in active use and development.

What's different

  • You can use Jinja {% include %} tags and {{ variables }} inside your Markdown content. This lets you insert constants and embed components inside your content. For example, I use it to insert a table of contents, calculators and constants in my content.
  • It transforms images and supports imgsrc and srcset out of the box, so responsive images are easy to implement. It can also create PDF thumbnails, and can be extended to transform other files.
  • You can build linters for your content. The default linters check if internal links and related entries exist. It's easy to write your own linters.
  • It's very extensible. You can add your own context processors, renderers and linters. You are not stuck with Markdown and Jinja. You can also create your own Markdown extensions.
  • It's relatively fast. All About Berlin and its hundreds of pages builds in 5 seconds on an M2 Macbook Air. Live reloads take around 400ms. It was originally built to run smoothly on a much older laptop.

Setup

Installation

Install Ursus with pip:

pip install ursus-ssg

Getting started

Call ursus to generate a static website. Call ursus --help to see the command line options it supports.

By default, Ursus looks for 3 directories, relative to the current directory:

  • It looks for content in ./content
  • It looks for page templates in ./templates
  • It generates a static website in ./output

For example, create a markdown file and save it as ./content/posts/first-post.md.

---
title: Hello world!
description: This is an example page
date_created: 2022-10-10
---

## Hello beautiful world

*This* is a template. Pretty cool eh?

Then, create a page template and save it as ./templates/posts/entry.html.jinja.

<!DOCTYPE html>
<html>
<head>
    <title>{{ entry.title }}</title>
    <meta name="description" content="{{ entry.description }}">
</head>
<body>
    {{ entry.body }}

    Created on {{ entry.date_created }}
</body>
</html>

Your project should now look like this:

my-website/ <- You are here
├─ content/
│  └─ posts/
│     └─ first-post.md
└─ templates/
   └─ posts/
      └─ entry.html.jinja

Call ursus to generate a statuc website. It will create ./output/posts/first-post.html.

Configuring Ursus

To configure Ursus, create a configuration file.

# Example Ursus config file
# Find all configuration options in `ursus/config.py`.
from ursus.config import config

config.content_path = Path(__file__).parent / 'blog'
config.templates_path = Path(__file__).parent / 'templates'
config.output_path = Path(__file__).parent.parent / 'dist'

config.site_url = 'https://allaboutberlin.com'

config.minify_js = True
config.minify_css = True

If you call your configuration file ursus_config.py, Ursus loads it automatically.

my-website/
├─ ursus_config.py
├─ content/
└─ templates/

You can also load a configuration file with the -w argument.

ursus -c /path/to/config.py

Watching for changes

Ursus can rebuild your website when the content or templates change.

# Rebuild when content or templates change
ursus -w
ursus --watch

It can only rebuild the pages that changed. This is much faster, but it does not work perfectly.

# Only rebuild the pages that changed
ursus -wf
ursus --watch --fast

Serving the website

Ursus can serve the website it generates. This is useful for testing.

# Serve the static website on port 80
ursus -s
ursus --serve 80

This is not meant for production. Use nginx, Caddy or some other static file server for that.

AI-translated multilingual site

Ursus can use OpenAI to translate your content to multiple languages. This feature is disabled by default. This tool is designed to only translate the parts of your content that change. It does not retranslate entire markdown files, but only the sections that changed.

1. Configure translations

Add these to your configuration:

# ursus_config.py

# Where .po, .pot and .mo translation files for your templates are stored. Optional.
config.translations_path = Path('...')

# OpenAI API key used to get translations from ChatGPT
config.openai_api_key = '...'

# The language of the original content
config.default_language = 'en'

# The desired translation languages
config.translation_languages = ['de', 'fr',]

# The Markdown fields you wish to translate. Other fields do not get translated.
config.metadata_fields_to_translate = ('title', 'short_title', 'description', )

2. Enable translations in your content

To translate Markdown files, add translation_* attributes to the metadata. The value is the desired URL of the translated page, relative to config.site_url.

---
title: How to change your address in Germany
short_title: How to change your address
description: ...
...
translation_de: de/ratgeber/adresswechsel.md
translation_fr: fr/guides/changement-dadresse.md
---

When you [move into a new apartment](/guides/moving-in), you must change your address. It's not automatic. This guide shows you how to do it.

In this example, the file guides/address-change.md would be turned into 3 entries in context['entries']:

  • context['entries']['en/guides/address-change.md']
  • context['entries']['de/ratgeber/adresswechsel.md']
  • context['entries']['fr/guides/changement-dadresse.md']

Each entry has a translations attribute that points to other entries. You can use it in your templates to link between entries.

context['entries']['en/guides/address-change.md'] == {
    'en': 'en/guides/address-change.md',
    'de': 'de/ratgeber/adresswechsel.md',
    'fr': 'fr/guides/changement-dadresse.md',
}

3. Render the entries

Your context now contains the original entry and its translations. If a template can render them, they will be rendered.

In the example above, you would need 3 templates:

  • /en/guides/address-change.html.jinja
  • /de/ratgeber/adresswechsel.html.jinja
  • /fr/guides/changement-dadresse.html.jinja

You can use Jinja includes to use the same template in all 3 places:

{# This is the content of de/ratgeber/entry.html.jinja #}
{% include "guides/entry.html.jinja" %}

4. Translate the templates

Jinja templates support localisation. Use {% trans %}your string here{% endtrans %} or {{ _('your string here') }}. Call ursus translate to generate gettext translation files, then call it again to compile your translation files.

How Ursus works

  1. Context processors generate the context used to render templates. The context is just a big dictionary that represent your site's entire content. Usually, each content file is turned into an entry.
  2. Renderers use the context and the templates to render the parts of the final website: pages, thumbnails, static assets, etc.

Content

Content is what fills your website: text, images, videos, PDFs. Content is usually rendered to create a working website. Some content (like Markdown files) is rendered with Templates, and other (like images) is converted to a different file format.

Ursus looks for content in ./content, unless you change config.content_path.

Entries

A single piece of content is called an Entry. This can be a single image, a single markdown file, etc.

Each Entry has a URI. This is the Entry's unique identifier. The URI is the Entry's path relative to the content directory. For example, the URI of ./content/posts/first-post.md is posts/first-post.md.

Context

The Context contains the information needed to render your website. It's just a big dictionary, and you can put anything in it.

context['entries'] contains is a dictionary of all your entries. The key is the Entry URI.

Context processors each add specific data to the context. For example, MarkdownProcessor adds your .md content to context.entries.

# Example context
{
    'entries': {
        'posts/first-post.md': {
            'title': 'Hello world!',
            'description': 'This is an example page',
            'date_created': datetime(2022, 10, 10),
            'body': '<h2>Hello beautiful world</h2><p>...',
        },
        'posts/second-post.md': {
            # ...
        },
    },
    # Context processors can add more things to the context
    'blog_title': 'Example blog',
    'site_url': 'https://example.com/blog',
}

Templates

Templates are used to render your Content. They are the theme of your website. Jinja templates, Javascript, CSS and theme images belong in the templates directory.

Ursus looks for templates in ./templates, unless you change config.templates_path.

Renderers

Renderers use the Context and the Templates to generate parts of your static website. For example, JinjaRenderer renders Jinja templates, ImageTransformRenderer converts and resizes your images, and StaticAssetRenderer copies your static assets.

Output

This is the final static website generated by Ursus. Ursus generates a static website in ./output, unless you change config.output_path.

The content of the output directory is ready to be served by any static file server.

How context processors work

Context processors transform the context, which is a dict with information about each of your Entries.

Context processors ignore file and directory names that start with . or _. For example, ./content/_drafts/hello.md and ./content/posts/_post-draft.md are ignored.

MarkdownProcessor

The MarkdownProcessor creates context for all .md files in content_path. The markdown content is in the body attribute.

{
    'entries': {
        'posts/first-post.md': {
            'title': 'Hello world!',
            'description': 'This is an example page',
            'date_created': datetime(2022, 10, 10),
            'body': '<h2>Hello beautiful world</h2><p>...',
        },
        # ...
    },
}

It makes a few changes to the default markdown output:

  • Put the front matter in the context
    • related_* keys are replaced by a list of related entry dicts
    • date_ keys are converted to datetime objects
    • Other attributes are added to the entry object.
  • Use responsive images based on config.image_transforms settings.
  • <img> are converted to <figure> or <picture> tags when appropriate.
  • Images are lazy-loaded with the loading=lazy attribute.
  • Jinja tags ({{ ... }} and {% ... %}) are rendered as-is. You can use {% include %} and {{ variables }} in your content.

GetEntriesProcessor

The GetEntriesProcessor adds a get_entries method to the context. It's used to get a list of entries of a certain type, and sort it.

{% set posts = get_entries('posts', filter_by=filter_function, sort_by='date_created', reverse=True) %}

{% for post in posts %}
...

GitDateProcessor

Adds the date_updated attribute to all Entries. It uses the file's last commit date.

{
    'entries': {
        'posts/first-post.md': {
            'date_updated': datetime(2022, 10, 10),
            # ...
        },
        # ...
    },
}

ImageProcessor

Adds images and PDFs Entries to the context. Dimensions and image transforms are added to each Entry. Use in combination with config.image_transforms.

{
    'entries': {
        'images/hello.jpg': {
            'width': 320,
            'height': 240,
            'image_transforms': [
                {
                    'is_default': True,
                    'input_mimetype': 'image/jpeg',
                    'output_mimetype': 'image/webp',
                    # ...
                },
                # ...
            ]
        },
        # ...
    },
}

How renderers work

Renderers use context and templates to generate parts of the static website.

A Generator takes your Content and your Templates and produces an Output. It's a recipe to turn your content into a final result. The default StaticSiteGenerator generates a static website. You can write your own Generator to output an eBook, a PDF, or anything else.

ImageTransformRenderer

Renders images in your content directory.

  • Images are converted and resized according to config.image_transforms.
  • Files that can't be transformed (PDF to PDF) are copied as-is to the output directory.
  • Images that can't be resized (SVG to anything) are copied as-is to the output directory.
  • Image EXIF data is removed.

This renderer does nothing unless config.image_transforms is set:

from ursus.config import config

config.image_transforms = {
    # ./content/images/test.jpg
    # ---> ./output/images/test.jpg
    # ./content/images/test.pdf
    # ---> ./output/images/test.pdf
    '': {
        'include': ('images/*', 'documents/*'),
        'output_types': ('original'),
    },
    # ./content/images/test.jpg
    # ---> ./output/images/content2x/test.jpg
    # ---> ./output/images/content2x/test.webp
    'content2x': {
        'include': ('images/*', 'illustrations/*'),
        'exclude': ('*.pdf', '*.svg'),
        'max_size': (800, 1200),
        'output_types': ('webp', 'original'),
    },
    # ./content/documents/test.pdf
    # ---> ./output/documents/pdfPreviews/test.png
    # ---> ./output/documents/pdfPreviews/test.webp
    'pdfPreviews': {
        'include': 'documents/*',
        'max_size': (300, 500),
        'output_types': ('webp', 'png'),
    },
}

JinjaRenderer

Renders *.jinja files in the templates directory.

The output file has the same name and relative path as the template, but the .jinja extension is removed.

my-website/
├─ templates/
│  ├─ contact.html.jinja
│  ├─ sitemap.xml.jinja
│  └─ posts/
│     └─ index.html.jinja
└─ output/
   ├─ contact.html
   ├─ sitemap.xml
   └─ posts/
      └─ index.html

Dedicated templates

Files named [name].[extension].jinja will render any entry with the same [name]. For example, first-post.md will render as first-post.txt using the template first-post.txt.jinja. It will also render as first-post.html using the template first-post.html.jinja.

my-website/
├─ content/
│  └─ posts/
│     └─ first-post.md
├─ templates/
│  └─ posts/
│     ├─ first-post.txt.jinja
│     └─ first-post.html.jinja
└─ output/
   └─ posts/
      ├─ first-post.txt
      └─ first-post.html

Entry templates

Files named entry.*.jinja will render every entry with the same relative path. For example, the template entry.html.jinja will be used to render first-post.md and second-post.md.

my-website/
├─ content/
│  └─ posts/
│     ├─ first-post.md
│     ├─ second-post.md
│     ├─ third-post.md
│     └─ _draft.md
├─ templates/
│  └─ posts/
│     ├─ entry.html.jinja
│     ├─ second-post.txt.jinja
│     └─ third-post.html.jinja
└─ output/
   └─ posts/
      ├─ first-post.html
      ├─ second-post.html
      ├─ second-post.txt  # Rendered with second-post.txt.jinja
      └─ third-post.html  # Rendered with third-post.html.jinja

Dedicated templates take precedence over entry templates with the same extension.

In the example above, third-post.md is rendered by third-post.html.jinja, not by entry.html.jinja. second-post.md is rendered by second-post.txt.jinja and by entry.html.jinja.

Ignored files

Files or directory names that start with . or _ are not rendered.

my-website/
├─ content/
│  └─ posts/
│     ├─ hello-world.md
│     ├─ .hidden.md
│     └─ _drafts
│        └─ not-rendered.md
├─ templates/
│  └─ posts/
│     └─ entry.html.jinja
└─ output/
   └─ posts/
      └─ hello-world.html

StaticAssetRenderer

Copies all files under ./templates except .jinja files to the same subdirectory in ./output. Files starting with . are ignored. Files and directories starting with _ are ignored.

my-website/
├─ templates/
│  ├─ _ignored.jpg
│  ├─ styles.css
│  ├─ images/
│  │  └─ hello.png
│  └─ js/
│     └─ test.js
└─ output/
   ├─ styles.css
   ├─ images/
   │  └─ hello.png
   └─ js/
      └─ test.js

It uses hard links instead of copying files, so it does not use extra disk space.

How generators work

Generators bring it all together. A generator takes all of your files, and generates some final product. There is only StaticSiteGenerator, which generates a static website. Custom generators could generate a book or a slideshow from the same content and templates.

How linters work

Ursus supports linter. They verify the content when ursus lint is called. You can find examples in ursus/linters.