From 1fdaf55406639555137a7b6dbb0d3a291fed9e51 Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Mon, 16 Oct 2023 19:37:31 -0500 Subject: [PATCH 01/10] Redirects (design doc): improving existing functionality Ref https://github.com/readthedocs/readthedocs.org/issues/4221 --- docs/dev/design/redirects.rst | 306 ++++++++++++++++++++++++++++++++++ 1 file changed, 306 insertions(+) create mode 100644 docs/dev/design/redirects.rst diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst new file mode 100644 index 00000000000..b6a7bd5d780 --- /dev/null +++ b/docs/dev/design/redirects.rst @@ -0,0 +1,306 @@ +Improving redirects +=================== + +Redirects are a core feature of Read the Docs, +they allow users to keep old URLs working when they rename or move a page. + +The current implementation lacks some features and has some undefined/undocumented behaviors. + +Goals +----- + +- Improve the user experience when creating redirects. +- Improve the current implementation without big breaking changes. + +No goals +-------- + +- Replicate every feature of other services without + having a clear use case for them. +- Improve the performance of redirects. + This can be discussed in an issue or pull request. +- Allow importing redirects. + We should push users to use our API instead. +- Allow specifying redirects in the RTD config file. + We have had several discussions around this, + but we haven't reached a consensus. + +Current implementation +---------------------- + +We have five types of redirects: + +Prefix redirect: + Allows to redirect all the URLs that start with a prefix to a new URL + using the default version and language of the project. + For example: a prefix redirect with the value ``/prefix/`` + will redirect ``/prefix/foo/bar`` to ``/en/latest/foo/bar``. + + They are basically the same as an exact redirect with a wildcard at the end. + They are a shortcut for a redirect like: + + - From: ``/prefix/$rest`` + To: ``/en/latest/`` + + Or maybe we could use a prefix redirect to replace the exact redirect with a wildcard? + +Page redirect: + Allows to redirect a single page to a new URL using the current version and language. + For example: a page redirect with the value ``/old/page.html`` + will redirect ``/en/latest/old/page.html`` to ``/en/latest/new/page.html``. + + Cross domain redirects are not allowed in page redirects. + They apply to all versions, + if you want it to apply only to a specific version you can use an exact redirect. + + A whole directory can't be redirected with a page redirect, + an exact redirect with a wildcard at the end needs to be used instead. + +Exact redirect: + Allows to redirect an exact URL to a new URL, + it allows a wildcard at the end to redirect. + For example: an exact redirect with the value ``/en/latest/page.html`` + will redirect ``/en/latest/page.html`` to the new URL. + + If an exact redirect with the value ``/en/latest/dir/$rest`` + is created, it will redirect all paths that start with ``/en/latest/dir/``, + the rest of the path will be added to the new URL automatically. + + - Cross domain redirects are allowed in exact redirects. + - They apply to all versions. + - A wildcard is allowed at the end of the URL. + - If a wildcard is used, the rest of the path will be added to the new URL automatically. + +Sphinx HTMLDir to HTML: + Allows to redirect clean-URLs to HTML URLs. + Useful in case a project changed the style of their URLs. + + They apply to all projects, not just Sphinx projects. + +Sphinx HTML to HTMLDir: + Allows to redirect HTML URLs to clean-URLs. + Useful in case a project changed the style of their URLs. + + They apply to all projects, not just Sphinx projects. + +How other services implement redirects +-------------------------------------- + +- Gitbook implementation is very basic, + they only allow page redirects. + + https://docs.gitbook.com/integrations/git-sync/content-configuration#redirects + +- Cloudflare pages allow to capture placeholders and one wildcard (in any part of the URL). + They also allow you to set the status code of the redirect, + and redirects can be specific in a ``_redirects`` file. + + https://developers.cloudflare.com/pages/platform/redirects/ + + They have a limit of 2100 redirects. + In case of multiple matches, the topmost redirect will be used. + +- Netlify allows to capture placeholders and a wildcard (only allowed at the end). + They also allow you to set the status code of the redirect, + and redirects can be specific in a ``_redirects`` file. + + - Forced redirects + - Match query arguments + - Match by country/language and cookies + - Per-domain and protocol redirects + - In case of multiple matches, the topmost redirect will be used. + + https://docs.netlify.com/routing/redirects/ + +Improving redirects +------------------- + +General improvements +~~~~~~~~~~~~~~~~~~~~ + +The following improvements will be applied to all types of redirects. + +- Allow choosing the status code of the redirect. + We already have a field for this, but it's not exposed to users. +- Allow to explicitly define the order of redirects. + This will be similar to the automation rules feature, + where users can reorder the rules so the most specific ones are first. + We currently rely on the implicit order of the redirects (updated_at). +- Allow to disable redirects. + It's useful when testing redirects, or when debugging a problem. + Instead of having to re-create the redirect, + we can just disable it and re-enable it later. +- Allow to add a short description. + It's useful to document why the redirect was created. + +Allow matching query arguments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We can do this in two ways: + +- At the DB level with some restrictions. + If done at the DB level, + we would need to have a different field + with just the path, and other with the query arguments normalized and sorted. + + For example, if we have a redirect with the value ``/foo?blue=1&yellow=2&red=3``, + if would be normalized in the DB as ``/foo`` and ``blue=1&red=3&yellow=2``. + This implies that the URL to be matched must have the exact same query arguments, + it can't have more or less. + + I believe the implementation described here is the same being used by Netlify, + since they have that same restriction. + + If the URL contains other parameters in addition to or instead of id, the request doesn't match that rule. + + https://docs.netlify.com/routing/redirects/redirect-options/#query-parameters + +- At the Python level. + If done at the DB level, + we would need to have a different field + with just the path, and other with query arguments. + + The matching of the path would be done at the DB level, + and the matching of the query arguments would be done at the Python level. + Here we can be more flexible, allowing any query arguments in the matched URL. + + We had some performance problems in the past, + but I believe it was mainly due to the use of regex instead of using string operations. + And matching the path is still done at the DB level. + We could limit the number of redirects that can be created with query arguments, + or the number of redirects in general. + +Don't run redirects on domains from pull request previews +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We currently run redirects on domains from pull request previews, +this is a problem when moving a whole project to a new domain. + +Do we have the need to run redirects on external domains? +They are suppose to be temporary domains. + +Normalize paths with trailing slashes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently, if users want to redirect a path with a trailing slash and without it, +they need to create two separate redirects (``/page/`` and ``/page``). + +We can simplify this by normalizing the path before matching it. + +For example: + +- From: ``/page/`` + To: ``/new/page`` + +The from path will be normalized to ``/page``, +and the filename to match will also be normalized before matching it. +This is similar to what Netlify does: +https://docs.netlify.com/routing/redirects/redirect-options/#trailing-slash. + +Page and exact redirects without a wildcard at the end will be normalized, +all other redirects need to be matched as is. + +Improving page redirects +~~~~~~~~~~~~~~~~~~~~~~~~ + +- Allow to redirect to external domains. + This can be useful to apply a redirect of a well known path + in all versions to another domain. + + For example, ``/security/`` to a their security policy page in another domain. + + This new feature isn't strictly needed, + but it will be useful to simplify the explanation of the feature + (one less restriction to explain). + +Improving exact redirects +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Explicitly place the ``$rest`` placeholder in the target URL, + instead of adding it automatically. + + Some times users want to redirect to a different path, + we have been adding a query parameter in the target URL to + prevent the old path from being added in the final path. + For example ``/new/path/?_=``. + + Instead of adding the path automatically, + users have to add the ``$rest`` placeholder in the target URL. + For example: + + - From: ``/old/path/$rest`` + To: ``/new/path/$rest`` + + - From: ``/old/path/$rest`` + To: ``/new/path/?page=$rest&foo=bar`` + +- Per-domain redirects. + Do users have the need for this? + The main problem is that we were applying the redirect + to external domains, if we stop doing that, is there the need for this? + We can also try to improve how our built-in redirects work + (specially our canonical domain redirect). + +Improving Sphinx redirects +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These redirects are useful, but we should rename them to something more general, +since they apply to all types of projects, not just Sphinx projects. + +Proposed names: + +- HTML URL to clean URL redirect (``file.html`` to ``file/``) +- Clean URL to HTML URL redirect (``file/`` to ``file.html``) + +Other ideas to improve redirects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Run forced redirects before built-in redirects. + We currently run built-in redirects before forced redirects, + this is a problem when moving a whole project to a new domain. + For example, a forced redirect like ``/$rest``, + won't work for the root URL of the project, + since ``/`` will first redirect to ``/en/latest/``. + + But shouldn't be a real problem, since users will still need to + handle the ``/en/latest/file/`` paths. + +- Run redirects on the edge. + Cloudflare allow us to create redirects on the edge, + but they have some limitations around the number of + redirect rules that can be created. + + And they will be useful for forced exact redirects only, + since we can't match a redirect based on the response of the origin server. + +- Merge prefix redirects with exact redirects. + Prefix redirects are the same as exact redirects with a wildcard at the end. + +- Placeholders. + I haven't seen users requesting this feature. + We can consider adding it in the future. + Maybe we can expose the current language and version as placeholders. + +- Replace ``$rest`` with ``*`` in the from_url. + This will be more consistent with other services, + but it will require users to re-learn the feature. + +- Per-protocol redirects. + We should push users to always use HTTPS. + +- Allow a prefix wildcard. + We currently only allow a suffix wildcard, + adding support for a prefix wildcard should be easy. + But do users need this feature? + +Migration +--------- + +Most of the proposed improvements are backwards compatible, +and just need a data migration to normalize existing redirects. + +For the exception of adding the ``$rest`` placeholder in the target URL explicitly, +that needs users to re-learn how this feature works, i.e, they may be expecting +to have the path added automatically in the target URL. + +We can create a small blog post explaining the changes. From fd067dcb624024bbbd558282ba507d957027c8a6 Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Mon, 16 Oct 2023 19:54:42 -0500 Subject: [PATCH 02/10] Update on page redirects --- docs/dev/design/redirects.rst | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index b6a7bd5d780..1e4b7e5f4eb 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -200,19 +200,6 @@ https://docs.netlify.com/routing/redirects/redirect-options/#trailing-slash. Page and exact redirects without a wildcard at the end will be normalized, all other redirects need to be matched as is. -Improving page redirects -~~~~~~~~~~~~~~~~~~~~~~~~ - -- Allow to redirect to external domains. - This can be useful to apply a redirect of a well known path - in all versions to another domain. - - For example, ``/security/`` to a their security policy page in another domain. - - This new feature isn't strictly needed, - but it will be useful to simplify the explanation of the feature - (one less restriction to explain). - Improving exact redirects ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -241,6 +228,27 @@ Improving exact redirects We can also try to improve how our built-in redirects work (specially our canonical domain redirect). +Improving page redirects +~~~~~~~~~~~~~~~~~~~~~~~~ + +- Allow to redirect to external domains. + This can be useful to apply a redirect of a well known path + in all versions to another domain. + + For example, ``/security/`` to a their security policy page in another domain. + + This new feature isn't strictly needed, + but it will be useful to simplify the explanation of the feature + (one less restriction to explain). + +- Allow a wildcard at the end of the from path. + This will allow users to migrate a whole directory to a new path + without having to create an exact redirect for each version. + + Similar to exact redirects, users need to add the ``$rest`` placeholder explicitly. + This means that that page redirects are the same as exact redirects, + with the only difference that they apply to all versions. + Improving Sphinx redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~ From 0412d21e9aebcfb291029938ae14f402626576e7 Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Mon, 16 Oct 2023 19:58:40 -0500 Subject: [PATCH 03/10] Local TOC --- docs/dev/design/redirects.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index 1e4b7e5f4eb..b34852885a4 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -6,6 +6,10 @@ they allow users to keep old URLs working when they rename or move a page. The current implementation lacks some features and has some undefined/undocumented behaviors. +.. contents:: + :local: + :depth: 3 + Goals ----- @@ -112,8 +116,8 @@ How other services implement redirects https://docs.netlify.com/routing/redirects/ -Improving redirects -------------------- +Improvements +------------ General improvements ~~~~~~~~~~~~~~~~~~~~ From 42805365077448d5f09764fe92fa64817d4e8d9f Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Mon, 23 Oct 2023 17:23:58 -0500 Subject: [PATCH 04/10] Small update --- docs/dev/design/redirects.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index b34852885a4..c9d0b5d97d2 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -16,8 +16,8 @@ Goals - Improve the user experience when creating redirects. - Improve the current implementation without big breaking changes. -No goals --------- +Non-goals +--------- - Replicate every feature of other services without having a clear use case for them. @@ -60,6 +60,8 @@ Page redirect: A whole directory can't be redirected with a page redirect, an exact redirect with a wildcard at the end needs to be used instead. + A page redirect on a single version project is the same as an exact redirect. + Exact redirect: Allows to redirect an exact URL to a new URL, it allows a wildcard at the end to redirect. From 983a92ae8d6d932634d615ba3dda34c8c1279252 Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Wed, 25 Oct 2023 12:05:34 -0500 Subject: [PATCH 05/10] Update with decisions --- docs/dev/design/redirects.rst | 141 +++++++++++++++++++--------------- 1 file changed, 77 insertions(+), 64 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index c9d0b5d97d2..ccd06223b1f 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -23,6 +23,7 @@ Non-goals having a clear use case for them. - Improve the performance of redirects. This can be discussed in an issue or pull request. + Performance should be considered when implementing new improvements. - Allow importing redirects. We should push users to use our API instead. - Allow specifying redirects in the RTD config file. @@ -43,8 +44,10 @@ Prefix redirect: They are basically the same as an exact redirect with a wildcard at the end. They are a shortcut for a redirect like: - - From: ``/prefix/$rest`` - To: ``/en/latest/`` + From: + ``/prefix/$rest`` + To: + ``/en/latest/`` Or maybe we could use a prefix redirect to replace the exact redirect with a wildcard? @@ -139,51 +142,14 @@ The following improvements will be applied to all types of redirects. - Allow to add a short description. It's useful to document why the redirect was created. -Allow matching query arguments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We can do this in two ways: - -- At the DB level with some restrictions. - If done at the DB level, - we would need to have a different field - with just the path, and other with the query arguments normalized and sorted. - - For example, if we have a redirect with the value ``/foo?blue=1&yellow=2&red=3``, - if would be normalized in the DB as ``/foo`` and ``blue=1&red=3&yellow=2``. - This implies that the URL to be matched must have the exact same query arguments, - it can't have more or less. - - I believe the implementation described here is the same being used by Netlify, - since they have that same restriction. - - If the URL contains other parameters in addition to or instead of id, the request doesn't match that rule. - - https://docs.netlify.com/routing/redirects/redirect-options/#query-parameters - -- At the Python level. - If done at the DB level, - we would need to have a different field - with just the path, and other with query arguments. - - The matching of the path would be done at the DB level, - and the matching of the query arguments would be done at the Python level. - Here we can be more flexible, allowing any query arguments in the matched URL. - - We had some performance problems in the past, - but I believe it was mainly due to the use of regex instead of using string operations. - And matching the path is still done at the DB level. - We could limit the number of redirects that can be created with query arguments, - or the number of redirects in general. - Don't run redirects on domains from pull request previews ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We currently run redirects on domains from pull request previews, this is a problem when moving a whole project to a new domain. -Do we have the need to run redirects on external domains? -They are suppose to be temporary domains. +We don't the need to run redirects on external domains, they +should be treated as temporary domains. Normalize paths with trailing slashes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -195,8 +161,10 @@ We can simplify this by normalizing the path before matching it. For example: -- From: ``/page/`` - To: ``/new/page`` +From: + ``/page/`` +To: + ``/new/page`` The from path will be normalized to ``/page``, and the filename to match will also be normalized before matching it. @@ -206,33 +174,32 @@ https://docs.netlify.com/routing/redirects/redirect-options/#trailing-slash. Page and exact redirects without a wildcard at the end will be normalized, all other redirects need to be matched as is. -Improving exact redirects -~~~~~~~~~~~~~~~~~~~~~~~~~ +This makes it impossible to match a path with a trailing slash. -- Explicitly place the ``$rest`` placeholder in the target URL, - instead of adding it automatically. +Explicit ``$rest`` placeholder +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Some times users want to redirect to a different path, - we have been adding a query parameter in the target URL to - prevent the old path from being added in the final path. - For example ``/new/path/?_=``. +Explicitly place the ``$rest`` placeholder in the target URL, +instead of adding it automatically. - Instead of adding the path automatically, - users have to add the ``$rest`` placeholder in the target URL. - For example: +Some times users want to redirect to a different path, +we have been adding a query parameter in the target URL to +prevent the old path from being added in the final path. +For example ``/new/path/?_=``. - - From: ``/old/path/$rest`` - To: ``/new/path/$rest`` +Instead of adding the path automatically, +users have to add the ``$rest`` placeholder in the target URL. +For example: - - From: ``/old/path/$rest`` - To: ``/new/path/?page=$rest&foo=bar`` +From: + ``/old/path/$rest`` +To: + ``/new/path/$rest`` -- Per-domain redirects. - Do users have the need for this? - The main problem is that we were applying the redirect - to external domains, if we stop doing that, is there the need for this? - We can also try to improve how our built-in redirects work - (specially our canonical domain redirect). +From: + ``/old/path/$rest`` +To: + ``/new/path/?page=$rest&foo=bar`` Improving page redirects ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -307,6 +274,52 @@ Other ideas to improve redirects adding support for a prefix wildcard should be easy. But do users need this feature? +- Per-domain redirects. + The main problem that originated this request was that we were applying redirects on external domains, + if we stop doing that, there is no need for this feature. + We can also try to improve how our built-in redirects work + (specially our canonical domain redirect). + +Allow matching query arguments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We can do this in two ways: + +- At the DB level with some restrictions. + If done at the DB level, + we would need to have a different field + with just the path, and other with the query arguments normalized and sorted. + + For example, if we have a redirect with the value ``/foo?blue=1&yellow=2&red=3``, + if would be normalized in the DB as ``/foo`` and ``blue=1&red=3&yellow=2``. + This implies that the URL to be matched must have the exact same query arguments, + it can't have more or less. + + I believe the implementation described here is the same being used by Netlify, + since they have that same restriction. + + If the URL contains other parameters in addition to or instead of id, the request doesn't match that rule. + + https://docs.netlify.com/routing/redirects/redirect-options/#query-parameters + +- At the Python level. + If done at the DB level, + we would need to have a different field + with just the path, and other with query arguments. + + The matching of the path would be done at the DB level, + and the matching of the query arguments would be done at the Python level. + Here we can be more flexible, allowing any query arguments in the matched URL. + + We had some performance problems in the past, + but I believe it was mainly due to the use of regex instead of using string operations. + And matching the path is still done at the DB level. + We could limit the number of redirects that can be created with query arguments, + or the number of redirects in general. + +We hava had only one user requesting this feature, +so this is not a priority. + Migration --------- From d943713efd1885d7ec9f2a72510cfc122789aee6 Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Wed, 25 Oct 2023 12:08:39 -0500 Subject: [PATCH 06/10] Comment about merging all redirect types --- docs/dev/design/redirects.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index ccd06223b1f..355cc6fa295 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -257,6 +257,11 @@ Other ideas to improve redirects - Merge prefix redirects with exact redirects. Prefix redirects are the same as exact redirects with a wildcard at the end. +- Merge all redirects into a single type. + This may simplify the implementation, + but it will make it harder to explain the feature to users. + And to replace some redirects we need to implement some new features. + - Placeholders. I haven't seen users requesting this feature. We can consider adding it in the future. From 9c73e5dfe4499e607136c319ec8a75a1583d7b1a Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Thu, 26 Oct 2023 13:02:10 -0500 Subject: [PATCH 07/10] GitLab pages + examples --- docs/dev/design/redirects.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index 355cc6fa295..87a9f4726cf 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -118,9 +118,21 @@ How other services implement redirects - Match by country/language and cookies - Per-domain and protocol redirects - In case of multiple matches, the topmost redirect will be used. + - Rewrites, serve a different file without redirecting. https://docs.netlify.com/routing/redirects/ +- GitLab pages supports the same syntax as Netlify, + and supports a subset of their features: + + - ``_redirects`` config file + - Status codes + - Rewrites + - Wildcards (splats) + - Placeholders + + https://docs.gitlab.com/ee/user/project/pages/redirects.html + Improvements ------------ @@ -214,6 +226,13 @@ Improving page redirects but it will be useful to simplify the explanation of the feature (one less restriction to explain). + Example: + + From: + ``/security/`` + To: + ``https://example.com/security/`` + - Allow a wildcard at the end of the from path. This will allow users to migrate a whole directory to a new path without having to create an exact redirect for each version. @@ -222,6 +241,13 @@ Improving page redirects This means that that page redirects are the same as exact redirects, with the only difference that they apply to all versions. + Example: + + From: + ``/old/path/$rest`` + To: + ``/new/path/$rest`` + Improving Sphinx redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~ From 68fa2f935267dc7fd0fd902b86c6e68984969c9b Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Thu, 26 Oct 2023 13:08:16 -0500 Subject: [PATCH 08/10] Normalize paths before saving them --- docs/dev/design/redirects.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index 87a9f4726cf..47bec7a86c8 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -169,7 +169,7 @@ Normalize paths with trailing slashes Currently, if users want to redirect a path with a trailing slash and without it, they need to create two separate redirects (``/page/`` and ``/page``). -We can simplify this by normalizing the path before matching it. +We can simplify this by normalizing the path before matching it, or before saving it. For example: From 9ebd216da9f15dd166a35949b5b6445d38c5906c Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Thu, 26 Oct 2023 15:36:04 -0500 Subject: [PATCH 09/10] Use * and :splat syntax --- docs/dev/design/redirects.rst | 40 ++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index 47bec7a86c8..a99ee748494 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -188,10 +188,22 @@ all other redirects need to be matched as is. This makes it impossible to match a path with a trailing slash. -Explicit ``$rest`` placeholder -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Use ``*`` and ``:splat`` for wildcards +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Explicitly place the ``$rest`` placeholder in the target URL, +Currently we are using ``$rest`` at the end of the ``From URL`` +to indicate that the rest of the path should be added to the target URL. + +A similar feature is implemented in other services using ``*`` and ``:splat``. + +Instead of using ``$rest`` in the URL for the suffix wildcard, we now will use ``*``, +and ``:splat`` as a placeholder in the target URL to be more consistent with other services. +Existing redirects can be migrated automatically. + +Explicit ``:splat`` placeholder +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Explicitly place the ``:splat`` placeholder in the target URL, instead of adding it automatically. Some times users want to redirect to a different path, @@ -200,18 +212,18 @@ prevent the old path from being added in the final path. For example ``/new/path/?_=``. Instead of adding the path automatically, -users have to add the ``$rest`` placeholder in the target URL. +users have to add the ``:splat`` placeholder in the target URL. For example: From: - ``/old/path/$rest`` + ``/old/path/*`` To: - ``/new/path/$rest`` + ``/new/path/:splat`` From: - ``/old/path/$rest`` + ``/old/path/*`` To: - ``/new/path/?page=$rest&foo=bar`` + ``/new/path/?page=:splat&foo=bar`` Improving page redirects ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -237,16 +249,16 @@ Improving page redirects This will allow users to migrate a whole directory to a new path without having to create an exact redirect for each version. - Similar to exact redirects, users need to add the ``$rest`` placeholder explicitly. + Similar to exact redirects, users need to add the ``:splat`` placeholder explicitly. This means that that page redirects are the same as exact redirects, with the only difference that they apply to all versions. Example: From: - ``/old/path/$rest`` + ``/old/path/*`` To: - ``/new/path/$rest`` + ``/new/path/:splat`` Improving Sphinx redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -262,6 +274,8 @@ Proposed names: Other ideas to improve redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The following improvements will not be implemented in the first iteration. + - Run forced redirects before built-in redirects. We currently run built-in redirects before forced redirects, this is a problem when moving a whole project to a new domain. @@ -293,10 +307,6 @@ Other ideas to improve redirects We can consider adding it in the future. Maybe we can expose the current language and version as placeholders. -- Replace ``$rest`` with ``*`` in the from_url. - This will be more consistent with other services, - but it will require users to re-learn the feature. - - Per-protocol redirects. We should push users to always use HTTPS. From 1afddab40d16d6c315d33250e09fa885eab5435e Mon Sep 17 00:00:00 2001 From: Santos Gallegos Date: Tue, 31 Oct 2023 10:21:49 -0500 Subject: [PATCH 10/10] Last updates --- docs/dev/design/redirects.rst | 36 +++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/docs/dev/design/redirects.rst b/docs/dev/design/redirects.rst index a99ee748494..40619ef3d12 100644 --- a/docs/dev/design/redirects.rst +++ b/docs/dev/design/redirects.rst @@ -260,6 +260,32 @@ Improving page redirects To: ``/new/path/:splat`` +Merge prefix redirects with exact redirects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Prefix redirects are the same as exact redirects with a wildcard at the end. +We will migrate all prefix redirects to exact redirects with a wildcard at the end. + +For example: + +From: + ``/prefix/`` + +Will be migrated to: + +From: + ``/prefix/*`` +To: + ``/en/latest/:splat`` + +Where ``/en/latest`` is the default version and language of the project. +For single version projects, the redirect will be: + +From: + ``/prefix/*`` +To: + ``/:splat`` + Improving Sphinx redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -294,9 +320,6 @@ The following improvements will not be implemented in the first iteration. And they will be useful for forced exact redirects only, since we can't match a redirect based on the response of the origin server. -- Merge prefix redirects with exact redirects. - Prefix redirects are the same as exact redirects with a wildcard at the end. - - Merge all redirects into a single type. This may simplify the implementation, but it will make it harder to explain the feature to users. @@ -324,7 +347,7 @@ The following improvements will not be implemented in the first iteration. Allow matching query arguments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We can do this in two ways: +We can do this in three ways: - At the DB level with some restrictions. If done at the DB level, @@ -343,6 +366,11 @@ We can do this in two ways: https://docs.netlify.com/routing/redirects/redirect-options/#query-parameters +- At the DB level using a JSONField. + All query arguments will be saved normalized as a dictionary. + When matching the URL, we will need to normalize the query arguments, + and use some a combination of ``has_keys`` and ``contained_by`` to match the exact number of query arguments. + - At the Python level. If done at the DB level, we would need to have a different field