Update GItlab from upstream, update pipelines and jobs schemas to inc…

…orporate all properties (#16) * feat: add streams for group and project variables (MeltanoLabs#64) * Add support for Group Variables and Project Variables * Update README * Add config for fetching group/project variables - defaults both to False * Update README * Fixup trailing commas Co-authored-by: Warren Ersly <[email protected]> Co-authored-by: Warren Ersly <[email protected]> * Update README.md * docs: add hyperlinks to main and legacy branches * feat: Add `ci_config_path` to projects schema (MeltanoLabs#97) * chore(deps): Bump requests from 2.20.0 to 2.31.0 (MeltanoLabs#99) Bumps [requests](https://github.com/psf/requests) from 2.20.0 to 2.31.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/psf/requests/releases">requests's releases</a>.</em></p> <blockquote> <h2>v2.31.0</h2> <h2>2.31.0 (2023-05-22)</h2> <p><strong>Security</strong></p> <ul> <li> <p>Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to potential forwarding of <code>Proxy-Authorization</code> headers to destination servers when following HTTPS redirects.</p> <p>When proxies are defined with user info (<a href="https://user:pass@proxy:8080">https://user:pass@proxy:8080</a>), Requests will construct a <code>Proxy-Authorization</code> header that is attached to the request to authenticate with the proxy.</p> <p>In cases where Requests receives a redirect response, it previously reattached the <code>Proxy-Authorization</code> header incorrectly, resulting in the value being sent through the tunneled connection to the destination server. Users who rely on defining their proxy credentials in the URL are <em>strongly</em> encouraged to upgrade to Requests 2.31.0+ to prevent unintentional leakage and rotate their proxy credentials once the change has been fully deployed.</p> <p>Users who do not use a proxy or do not supply their proxy credentials through the user information portion of their proxy URL are not subject to this vulnerability.</p> <p>Full details can be read in our <a href="https://github.com/psf/requests/security/advisories/GHSA-j8r2-6x86-q33q">Github Security Advisory</a> and <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-32681">CVE-2023-32681</a>.</p> </li> </ul> <h2>v2.30.0</h2> <h2>2.30.0 (2023-05-03)</h2> <p><strong>Dependencies</strong></p> <ul> <li> <p>⚠️ Added support for urllib3 2.0. ⚠️</p> <p>This may contain minor breaking changes so we advise careful testing and reviewing <a href="https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html">https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html</a> prior to upgrading.</p> <p>Users who wish to stay on urllib3 1.x can pin to <code>urllib3<2</code>.</p> </li> </ul> <h2>v2.29.0</h2> <h2>2.29.0 (2023-04-26)</h2> <p><strong>Improvements</strong></p> <ul> <li>Requests now defers chunked requests to the urllib3 implementation to improve standardization. (<a href="https://redirect.github.com/psf/requests/issues/6226">#6226</a>)</li> <li>Requests relaxes header component requirements to support bytes/str subclasses. (<a href="https://redirect.github.com/psf/requests/issues/6356">#6356</a>)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psf/requests/blob/main/HISTORY.md">requests's changelog</a>.</em></p> <blockquote> <h2>2.31.0 (2023-05-22)</h2> <p><strong>Security</strong></p> <ul> <li> <p>Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to potential forwarding of <code>Proxy-Authorization</code> headers to destination servers when following HTTPS redirects.</p> <p>When proxies are defined with user info (<a href="https://user:pass@proxy:8080">https://user:pass@proxy:8080</a>), Requests will construct a <code>Proxy-Authorization</code> header that is attached to the request to authenticate with the proxy.</p> <p>In cases where Requests receives a redirect response, it previously reattached the <code>Proxy-Authorization</code> header incorrectly, resulting in the value being sent through the tunneled connection to the destination server. Users who rely on defining their proxy credentials in the URL are <em>strongly</em> encouraged to upgrade to Requests 2.31.0+ to prevent unintentional leakage and rotate their proxy credentials once the change has been fully deployed.</p> <p>Users who do not use a proxy or do not supply their proxy credentials through the user information portion of their proxy URL are not subject to this vulnerability.</p> <p>Full details can be read in our <a href="https://github.com/psf/requests/security/advisories/GHSA-j8r2-6x86-q33q">Github Security Advisory</a> and <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-32681">CVE-2023-32681</a>.</p> </li> </ul> <h2>2.30.0 (2023-05-03)</h2> <p><strong>Dependencies</strong></p> <ul> <li> <p>⚠️ Added support for urllib3 2.0. ⚠️</p> <p>This may contain minor breaking changes so we advise careful testing and reviewing <a href="https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html">https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html</a> prior to upgrading.</p> <p>Users who wish to stay on urllib3 1.x can pin to <code>urllib3<2</code>.</p> </li> </ul> <h2>2.29.0 (2023-04-26)</h2> <p><strong>Improvements</strong></p> <ul> <li>Requests now defers chunked requests to the urllib3 implementation to improve standardization. (<a href="https://redirect.github.com/psf/requests/issues/6226">#6226</a>)</li> <li>Requests relaxes header component requirements to support bytes/str subclasses. (<a href="https://redirect.github.com/psf/requests/issues/6356">#6356</a>)</li> </ul> <h2>2.28.2 (2023-01-12)</h2>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psf/requests/commit/147c8511ddbfa5e8f71bbf5c18ede0c4ceb3bba4"><code>147c851</code></a> v2.31.0</li> <li><a href="https://github.com/psf/requests/commit/74ea7cf7a6a27a4eeb2ae24e162bcc942a6706d5"><code>74ea7cf</code></a> Merge pull request from GHSA-j8r2-6x86-q33q</li> <li><a href="https://github.com/psf/requests/commit/302225334678490ec66b3614a9dddb8a02c5f4fe"><code>3022253</code></a> test on pypy 3.8 and pypy 3.9 on windows and macos (<a href="https://redirect.github.com/psf/requests/issues/6424">#6424</a>)</li> <li><a href="https://github.com/psf/requests/commit/b639e66c816514e40604d46f0088fbceec1a5149"><code>b639e66</code></a> test on py3.12 (<a href="https://redirect.github.com/psf/requests/issues/6448">#6448</a>)</li> <li><a href="https://github.com/psf/requests/commit/d3d504436ef0c2ac7ec8af13738b04dcc8c694be"><code>d3d5044</code></a> Fixed a small typo (<a href="https://redirect.github.com/psf/requests/issues/6452">#6452</a>)</li> <li><a href="https://github.com/psf/requests/commit/2ad18e0e10e7d7ecd5384c378f25ec8821a10a29"><code>2ad18e0</code></a> v2.30.0</li> <li><a href="https://github.com/psf/requests/commit/f2629e9e3c7ce3c3c8c025bcd8db551101cbc773"><code>f2629e9</code></a> Remove strict parameter (<a href="https://redirect.github.com/psf/requests/issues/6434">#6434</a>)</li> <li><a href="https://github.com/psf/requests/commit/87d63de8739263bbe17034fba2285c79780da7e8"><code>87d63de</code></a> v2.29.0</li> <li><a href="https://github.com/psf/requests/commit/51716c4ef390136b0d4b800ec7665dd5503e64fc"><code>51716c4</code></a> enable the warnings plugin (<a href="https://redirect.github.com/psf/requests/issues/6416">#6416</a>)</li> <li><a href="https://github.com/psf/requests/commit/a7da1ab3498b10ec3a3582244c94b2845f8a8e71"><code>a7da1ab</code></a> try on ubuntu 22.04 (<a href="https://redirect.github.com/psf/requests/issues/6418">#6418</a>)</li> <li>Additional commits viewable in <a href="https://github.com/psf/requests/compare/v2.20.0...v2.31.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=requests&package-manager=pip&previous-version=2.20.0&new-version=2.31.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/MeltanoLabs/tap-gitlab/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Managed by Pulumi * Managed by Pulumi * chore(deps): Bump requests from 2.31.0 to 2.32.0 (MeltanoLabs#101) Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/psf/requests/releases">requests's releases</a>.</em></p> <blockquote> <h2>v2.32.0</h2> <h2>2.32.0 (2024-05-20)</h2> <h2>🐍 PYCON US 2024 EDITION 🐍</h2> <p><strong>Security</strong></p> <ul> <li>Fixed an issue where setting <code>verify=False</code> on the first request from a Session will cause subsequent requests to the <em>same origin</em> to also ignore cert verification, regardless of the value of <code>verify</code>. (<a href="https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56">https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56</a>)</li> </ul> <p><strong>Improvements</strong></p> <ul> <li><code>verify=True</code> now reuses a global SSLContext which should improve request time variance between first and subsequent requests. It should also minimize certificate load time on Windows systems when using a Python version built with OpenSSL 3.x. (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li> <li>Requests now supports optional use of character detection (<code>chardet</code> or <code>charset_normalizer</code>) when repackaged or vendored. This enables <code>pip</code> and other projects to minimize their vendoring surface area. The <code>Response.text()</code> and <code>apparent_encoding</code> APIs will default to <code>utf-8</code> if neither library is present. (<a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a>)</li> </ul> <p><strong>Bugfixes</strong></p> <ul> <li>Fixed bug in length detection where emoji length was incorrectly calculated in the request content-length. (<a href="https://redirect.github.com/psf/requests/issues/6589">#6589</a>)</li> <li>Fixed deserialization bug in JSONDecodeError. (<a href="https://redirect.github.com/psf/requests/issues/6629">#6629</a>)</li> <li>Fixed bug where an extra leading <code>/</code> (path separator) could lead urllib3 to unnecessarily reparse the request URI. (<a href="https://redirect.github.com/psf/requests/issues/6644">#6644</a>)</li> </ul> <p><strong>Deprecations</strong></p> <ul> <li>Requests has officially added support for CPython 3.12 (<a href="https://redirect.github.com/psf/requests/issues/6503">#6503</a>)</li> <li>Requests has officially added support for PyPy 3.9 and 3.10 (<a href="https://redirect.github.com/psf/requests/issues/6641">#6641</a>)</li> <li>Requests has officially dropped support for CPython 3.7 (<a href="https://redirect.github.com/psf/requests/issues/6642">#6642</a>)</li> <li>Requests has officially dropped support for PyPy 3.7 and 3.8 (<a href="https://redirect.github.com/psf/requests/issues/6641">#6641</a>)</li> </ul> <p><strong>Documentation</strong></p> <ul> <li>Various typo fixes and doc improvements.</li> </ul> <p><strong>Packaging</strong></p> <ul> <li>Requests has started adopting some modern packaging practices. The source files for the projects (formerly <code>requests</code>) is now located in <code>src/requests</code> in the Requests sdist. (<a href="https://redirect.github.com/psf/requests/issues/6506">#6506</a>)</li> <li>Starting in Requests 2.33.0, Requests will migrate to a PEP 517 build system using <code>hatchling</code>. This should not impact the average user, but extremely old versions of packaging utilities may have issues with the new packaging format.</li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/matthewarmand"><code>@matthewarmand</code></a> made their first contribution in <a href="https://redirect.github.com/psf/requests/pull/6258">psf/requests#6258</a></li> <li><a href="https://github.com/cpzt"><code>@cpzt</code></a> made their first contribution in <a href="https://redirect.github.com/psf/requests/pull/6456">psf/requests#6456</a></li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psf/requests/blob/main/HISTORY.md">requests's changelog</a>.</em></p> <blockquote> <h2>2.32.0 (2024-05-20)</h2> <p><strong>Security</strong></p> <ul> <li>Fixed an issue where setting <code>verify=False</code> on the first request from a Session will cause subsequent requests to the <em>same origin</em> to also ignore cert verification, regardless of the value of <code>verify</code>. (<a href="https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56">https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56</a>)</li> </ul> <p><strong>Improvements</strong></p> <ul> <li><code>verify=True</code> now reuses a global SSLContext which should improve request time variance between first and subsequent requests. It should also minimize certificate load time on Windows systems when using a Python version built with OpenSSL 3.x. (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li> <li>Requests now supports optional use of character detection (<code>chardet</code> or <code>charset_normalizer</code>) when repackaged or vendored. This enables <code>pip</code> and other projects to minimize their vendoring surface area. The <code>Response.text()</code> and <code>apparent_encoding</code> APIs will default to <code>utf-8</code> if neither library is present. (<a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a>)</li> </ul> <p><strong>Bugfixes</strong></p> <ul> <li>Fixed bug in length detection where emoji length was incorrectly calculated in the request content-length. (<a href="https://redirect.github.com/psf/requests/issues/6589">#6589</a>)</li> <li>Fixed deserialization bug in JSONDecodeError. (<a href="https://redirect.github.com/psf/requests/issues/6629">#6629</a>)</li> <li>Fixed bug where an extra leading <code>/</code> (path separator) could lead urllib3 to unnecessarily reparse the request URI. (<a href="https://redirect.github.com/psf/requests/issues/6644">#6644</a>)</li> </ul> <p><strong>Deprecations</strong></p> <ul> <li>Requests has officially added support for CPython 3.12 (<a href="https://redirect.github.com/psf/requests/issues/6503">#6503</a>)</li> <li>Requests has officially added support for PyPy 3.9 and 3.10 (<a href="https://redirect.github.com/psf/requests/issues/6641">#6641</a>)</li> <li>Requests has officially dropped support for CPython 3.7 (<a href="https://redirect.github.com/psf/requests/issues/6642">#6642</a>)</li> <li>Requests has officially dropped support for PyPy 3.7 and 3.8 (<a href="https://redirect.github.com/psf/requests/issues/6641">#6641</a>)</li> </ul> <p><strong>Documentation</strong></p> <ul> <li>Various typo fixes and doc improvements.</li> </ul> <p><strong>Packaging</strong></p> <ul> <li>Requests has started adopting some modern packaging practices. The source files for the projects (formerly <code>requests</code>) is now located in <code>src/requests</code> in the Requests sdist. (<a href="https://redirect.github.com/psf/requests/issues/6506">#6506</a>)</li> <li>Starting in Requests 2.33.0, Requests will migrate to a PEP 517 build system using <code>hatchling</code>. This should not impact the average user, but extremely old versions of packaging utilities may have issues with the new packaging format.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psf/requests/commit/d6ebc4a2f1f68b7e355fb7e4dd5ffc0845547f9f"><code>d6ebc4a</code></a> v2.32.0</li> <li><a href="https://github.com/psf/requests/commit/9a40d1277807f0a4f26c9a37eea8ec90faa8aadc"><code>9a40d12</code></a> Avoid reloading root certificates to improve concurrent performance (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li> <li><a href="https://github.com/psf/requests/commit/0c030f78d24f29a459dbf39b28b4cc765e2153d7"><code>0c030f7</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a> from nateprewitt/no_char_detection</li> <li><a href="https://github.com/psf/requests/commit/555b870eb19d497ddb67042645420083ec8efb02"><code>555b870</code></a> Allow character detection dependencies to be optional in post-packaging steps</li> <li><a href="https://github.com/psf/requests/commit/d6dded3f00afcf56a7e866cb0732799045301eb0"><code>d6dded3</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6700">#6700</a> from franekmagiera/update-redirect-to-invalid-uri-test</li> <li><a href="https://github.com/psf/requests/commit/bf24b7d8d17da34be720c19e5978b2d3bf94a53b"><code>bf24b7d</code></a> Use an invalid URI that will not cause httpbin to throw 500</li> <li><a href="https://github.com/psf/requests/commit/2d5f54779ad174035c5437b3b3c1146b0eaf60fe"><code>2d5f547</code></a> Pin 3.8 and 3.9 runners back to macos-13 (<a href="https://redirect.github.com/psf/requests/issues/6688">#6688</a>)</li> <li><a href="https://github.com/psf/requests/commit/f1bb07d39b74d6444e333879f8b8a3d9dd4d2311"><code>f1bb07d</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6687">#6687</a> from psf/dependabot/github_actions/github/codeql-act...</li> <li><a href="https://github.com/psf/requests/commit/60047ade64b0b882cbc94e047198818ab580911e"><code>60047ad</code></a> Bump github/codeql-action from 3.24.0 to 3.25.0</li> <li><a href="https://github.com/psf/requests/commit/31ebb8102c00f8cf8b396a6356743cca4362e07b"><code>31ebb81</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6682">#6682</a> from frenzymadness/pytest8</li> <li>Additional commits viewable in <a href="https://github.com/psf/requests/compare/v2.31.0...v2.32.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=requests&package-manager=pip&previous-version=2.31.0&new-version=2.32.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/MeltanoLabs/tap-gitlab/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: add on_predicate decorator to handle 429 (MeltanoLabs#102) ## Description This MR aims to cover the case HTTP `429` "Too Many Requests" ([RFC685](https://www.rfc-editor.org/rfc/rfc6585#section-4))i n a more specific way rather than the current way to handle any exception between the HTTP status code 400 and 500. This is motivated by the issue opened by @kgpayne [here](MeltanoLabs#94). ## Implementation The first decorator to be executed and added on top will be only evaluated when the HTTP code is 429 and then backoff whatever time is in the `Retry-After` header that gitlab returns (see [here](https://docs.gitlab.com/ee/administration/settings/user_and_ip_rate_limits.html#response-headers)). The example is literally an implementation of the example supplied in the README.md from the backoff library [here](https://github.com/litl/backoff?tab=readme-ov-file#backoffruntime) + adding the same amount of max_times. --------- Co-authored-by: Alejandro Martinez <[email protected]> Co-authored-by: Edgar Ramírez Mondragón <[email protected]> * fix: Bump `backoff` to 2.2.1 (MeltanoLabs#103) Follows from MeltanoLabs#102 * fix: Bump singer-python to 6.0.1 (MeltanoLabs#104) Related: * MeltanoLabs#102 * MeltanoLabs#103 * Fix things after merge * Update schemas for pipelines and jobs --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: wersly <[email protected]> Co-authored-by: Warren Ersly <[email protected]> Co-authored-by: Warren Ersly <[email protected]> Co-authored-by: Aaron ("AJ") Steers <[email protected]> Co-authored-by: Francis Potter <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: MeltyBot <[email protected]> Co-authored-by: Alejandro Martínez Otal <[email protected]> Co-authored-by: Alejandro Martinez <[email protected]> Co-authored-by: Edgar Ramírez Mondragón <[email protected]>
minwareco · Jun 27, 2024 · f2125be · f2125be
1 parent 1c2bdd4
commit f2125be
Show file tree

Hide file tree

Showing 9 changed files with 602 additions and 124 deletions.
diff --git a/README.md b/README.md
@@ -2,6 +2,14 @@
 
 This is a [Singer](https://singer.io) tap that produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
 
+## Important notice
+
+- This repository's default branch [`legacy-stable`](https://github.com/MeltanoLabs/tap-gitlab/tree/legacy-stable) is kept for compatibility reasons but is no longer under active development.
+- New development is being performed against the [`main`](https://github.com/MeltanoLabs/tap-gitlab/tree/main) branch, which is based on a port to the Meltano SDK in Pull Request #65.
+- For a stable experience, users of this tap should begin pinning their installations to a specific [release](https://github.com/MeltanoLabs/tap-gitlab/releases) instead of branch references. More instructions are provided within the `README.md` of the `main` branch.
+
+## About this tap
+
 It is based on v0.5.1 of <https://github.com/singer-io/tap-gitlab>, but contains [many additional improvements](./CHANGELOG.md).
 
 This tap:
@@ -26,6 +34,8 @@ This tap:
   - [Epics](https://docs.gitlab.com/ee/api/epics.html) (only available for GitLab Ultimate and GitLab.com Gold accounts)
   - [Epic Issues](https://docs.gitlab.com/ee/api/epic_issues.html) (only available for GitLab Ultimate and GitLab.com Gold accounts)
   - [Vulnerabilities](https://docs.gitlab.com/ee/api/project_vulnerabilities.html)
+  - [Group Variables](https://docs.gitlab.com/ee/api/group_level_variables.html)
+  - [Project Variables](https://docs.gitlab.com/ee/api/project_level_variables.html)
 - Outputs the schema for each resource
 - Incrementally pulls data based on the input state
 
@@ -68,7 +78,9 @@ pip install git+https://gitlab.com/meltano/tap-gitlab.git
       "start_date": "2018-01-01T00:00:00Z",
       "ultimate_license": true,
       "fetch_merge_request_commits": false,
-      "fetch_pipelines_extended": false
+      "fetch_pipelines_extended": false,
+      "fetch_group_variables": false,
+      "fetch_project_variables": false
     }
     ```
 
@@ -80,6 +92,10 @@ pip install git+https://gitlab.com/meltano/tap-gitlab.git
 
     If `fetch_pipelines_extended` is true (defaults to false), then for every Pipeline fetched with `sync_pipelines` (which returns N pages containing all pipelines per project), also fetch extended details of each of these pipelines with `sync_pipelines_extended`. Similar concerns as those related to `fetch_merge_request_commits` apply here - every pipeline fetched with `sync_pipelines_extended` requires a separate API call.
 
+    If `fetch_group_variables` is true (defaults to false), then Group-level CI/CD variables will be retrieved for each available / specified group. This feature is treated as an opt-in to prevent users from accidentally extracting any potential secrets stored as Group-level CI/CD variables.
+
+    If `fetch_project_variables` is true (defaults to false), then Project-level CI/CD variables will be retrieved for each available / specified project. This feature is treated as an opt-in to prevent users from accidentally extracting any potential secrets stored as Project-level CI/CD variables.
+
 4. [Optional] Create the initial state file
 
     You can provide JSON file that contains a date for the API endpoints

diff --git a/setup.py b/setup.py
@@ -11,10 +11,10 @@
       classifiers=['Programming Language :: Python :: 3 :: Only'],
       py_modules=['tap_gitlab'],
       install_requires=[
-          'singer-python==5.9.1',
-          'requests==2.20.0',
+          'singer-python==6.0.1',
+          'requests==2.32.0',
           'strict-rfc3339==0.7',
-          'backoff==1.8.0',
+          'backoff==2.2.1',
           'psutil==5.8.0',
           'gitlocal@git+https://{}@github.com/minwareco/gitlocal.git'.format(os.environ.get("GITHUB_TOKEN", ""))
       ],
@@ -40,6 +40,8 @@
             "tags.json",
             "releases.json",
             "vulnerabilities.json",
+            "project_variables.json",
+            "group_variables.json"
           ],
       },
       include_package_data=True,

diff --git a/tap_gitlab/__init__.py b/tap_gitlab/__init__.py
@@ -29,7 +29,12 @@
     'private_token': None,
     'start_date': None,
     'groups': '',
-    'ultimate_license': False
+    'ultimate_license': False,
+    'fetch_merge_request_commits': False,
+    'fetch_merge_request_notes': False,
+    'fetch_pipelines_extended': False,
+    'fetch_group_variables': False,
+    'fetch_project_variables': False,
 }
 STATE = {}
 CATALOG = None
@@ -132,6 +137,7 @@ def load_schema(entity):
         'url': '/groups/{id}/subgroups',
         'schema': load_schema('groups'),
         'key_properties': ['id'],
+        'replication_method': 'FULL_TABLE',
     },
     'group_milestones': {
         'url': '/groups/{id}/milestones',
@@ -225,16 +231,34 @@ def load_schema(entity):
         'key_properties': ['id'],
         'replication_method': 'FULL_TABLE',
     },
-     'vulnerabilities': {
-            'url': '/projects/{id}/vulnerabilities',
-            'schema': load_schema('vulnerabilities'),
-            'key_properties': ['id'],
-            'replication_method': 'FULL_TABLE',
-        },
+    'vulnerabilities': {
+        'url': '/projects/{id}/vulnerabilities',
+        'schema': load_schema('vulnerabilities'),
+        'key_properties': ['id'],
+        'replication_method': 'FULL_TABLE',
+    },
+    'project_variables': {
+        'url': '/projects/{id}/variables',
+        'schema': load_schema('project_variables'),
+        'key_properties': ['project_id', 'key'],
+        'replication_method': 'FULL_TABLE',
+    },
+    'group_variables': {
+        'url': '/groups/{id}/variables',
+        'schema': load_schema('group_variables'),
+        'key_properties': ['group_id', 'key'],
+        'replication_method': 'FULL_TABLE',
+    }
 }
 
 ULTIMATE_RESOURCES = ("epics", "epic_issues")
-STREAM_CONFIG_SWITCHES = ('merge_request_commits', 'merge_request_notes', 'pipelines_extended')
+STREAM_CONFIG_SWITCHES = (
+    'merge_request_commits',
+    'merge_request_notes',
+    'pipelines_extended',
+    'group_variables',
+    'project_variables',
+)
 
 LOGGER = singer.get_logger()
 SESSION = requests.Session()
@@ -276,17 +300,15 @@ def get_start(entity):
         STATE[entity] = max(dates_to_compare).isoformat()
     return STATE[entity]
 
-
-# TODO : when singer-python updates the backoff module
-# we should update this to pull the exact time to wait from the header
-@backoff.on_predicate(backoff.expo,
-                      lambda x: x.status_code == 429,
-                      max_tries=10,
-                      jitter=backoff.random_jitter)
+@backoff.on_predicate(backoff.runtime,
+                      predicate=lambda r: r.status_code == 429,
+                      max_tries=5, 
+                      value=lambda r: int(r.headers.get("Retry-After")), 
+                      jitter=None)
 @backoff.on_exception(backoff.expo,
                       (requests.exceptions.RequestException),
                       max_tries=5,
-                      giveup=lambda e: e.response is not None and 400 <= e.response.status_code < 500, # pylint: disable=line-too-long
+                      giveup=lambda e: e.response is not None and e.response.status_code != 429 and 400 <= e.response.status_code < 500, # pylint: disable=line-too-long
                       factor=2)
 def request(url, params=None):
     params = params or {}
@@ -307,12 +329,6 @@ def request(url, params=None):
         LOGGER.info("Skipping request to {}".format(url))
         LOGGER.info("Reason: {} - {}".format(resp.status_code, resp.content))
         raise ResourceInaccessible
-    # if we are being rate limited, let the backoff logic run
-    elif resp.status_code != 429 and resp.status_code >= 400:
-        LOGGER.critical(
-            "Error making request to GitLab API: GET {} [{} - {}]".format(
-                url, resp.status_code, resp.content))
-        sys.exit(1)
 
     return resp
 
@@ -1004,6 +1020,8 @@ def sync_group(gid, pids, gitLocal):
 
     sync_labels(data, "group")
 
+    sync_variables(data, "group")
+
     if CONFIG['ultimate_license']:
         sync_epics(data)
 
@@ -1017,6 +1035,8 @@ def sync_group(gid, pids, gitLocal):
 def sync_pipelines(project):
     entity = "pipelines"
     stream = CATALOG.get_stream(entity)
+
+    LOGGER.info('Stream Pipelines: {}'.format(stream.is_selected()))
     if stream is None or not stream.is_selected():
         return
 
@@ -1030,7 +1050,12 @@ def sync_pipelines(project):
     with Transformer(pre_hook=format_timestamp) as transformer:
         for row in gen_request(url):
 
-            transformed_row = transformer.transform(row, RESOURCES[entity]["schema"], mdata)
+            pipeline_record = {
+                **row,
+                'project_id': project['id'],
+                '_sdc_repository': project['path_with_namespace']
+            }
+            transformed_row = transformer.transform(pipeline_record, RESOURCES[entity]["schema"], mdata)
 
             # Write the Pipeline record
             singer.write_record(entity, transformed_row, time_extracted=utils.now())
@@ -1058,8 +1083,12 @@ def sync_pipelines_extended(project, pipeline):
 
     with Transformer(pre_hook=format_timestamp) as transformer:
         for row in gen_request(url):
-            row['project_id'] = project['id']
-            transformed_row = transformer.transform(row, RESOURCES[entity]["schema"], mdata)
+            pipeline_extended_record = {
+                **row,
+                'project_id': project['id'],
+                '_sdc_repository': project['path_with_namespace']
+            }
+            transformed_row = transformer.transform(pipeline_extended_record, RESOURCES[entity]["schema"], mdata)
 
             singer.write_record(entity, transformed_row, time_extracted=utils.now())
 
@@ -1088,13 +1117,13 @@ def sync_jobs(project, pipeline):
     url = get_url(entity=entity, id=project['id'], secondary_id=pipeline['id'])
     with Transformer(pre_hook=format_timestamp) as transformer:
         for row in gen_request(url):
-            row['project_id'] = project['id']
-            flatten_id(row, 'user')
-            flatten_id(row, 'commit')
-            flatten_id(row, 'pipeline')
-            flatten_id(row, 'runner')
-
-            transformed_row = transformer.transform(row, RESOURCES[entity]['schema'], mdata)
+            job_record = {
+                **row,
+                'project_id': project['id'],
+                'pipeline_id': pipeline['id'],
+                '_sdc_repository': project['path_with_namespace']
+            }
+            transformed_row = transformer.transform(job_record, RESOURCES[entity]['schema'], mdata)
             singer.write_record(entity, transformed_row, time_extracted=utils.now())
 
 def write_repository(raw_repo):
@@ -1123,6 +1152,21 @@ def write_repository(raw_repo):
         rec = transformer.transform(repo, Schema.to_dict(stream.schema), metadata=metadata.to_map(stream.metadata))
     singer.write_record('repositories', rec, time_extracted=extraction_time)
 
+def sync_variables(entity, element="project"):
+    stream_name = "{}_variables".format(element)
+    stream = CATALOG.get_stream(stream_name)
+    if stream is None or not stream.is_selected():
+        return
+    mdata = metadata.to_map(stream.metadata)
+
+    url = get_url(entity=element + "_variables", id=entity['id'])
+
+    with Transformer(pre_hook=format_timestamp) as transformer:
+        for row in gen_request(url):
+            row[element + '_id'] = entity['id']
+            transformed_row = transformer.transform(row, RESOURCES[element + "_variables"]["schema"], mdata)
+            singer.write_record(element + "_variables", transformed_row, time_extracted=utils.now())
+
 def sync_project(pid, gitLocal):
     url = get_url(entity="projects", id=pid)
 
@@ -1190,6 +1234,8 @@ def sync_project(pid, gitLocal):
         sync_tags(data)
         sync_pipelines(data)
         sync_vulnerabilities(data)
+        sync_variables(data)
+
 
 def do_sync():
     LOGGER.info("Starting sync")
@@ -1285,6 +1331,11 @@ def main_impl():
 
     CONFIG.update(args.config)
     CONFIG['ultimate_license'] = truthy(CONFIG['ultimate_license'])
+    CONFIG['fetch_merge_request_commits'] = truthy(CONFIG['fetch_merge_request_commits'])
+    CONFIG['fetch_merge_request_notes'] = truthy(CONFIG['fetch_merge_request_notes'])
+    CONFIG['fetch_pipelines_extended'] = truthy(CONFIG['fetch_pipelines_extended'])
+    CONFIG['fetch_group_variables'] = truthy(CONFIG['fetch_group_variables'])
+    CONFIG['fetch_project_variables'] = truthy(CONFIG['fetch_project_variables'])
 
     if '/api/' not in CONFIG['api_url']:
         CONFIG['api_url'] += '/api/v4'

diff --git a/tap_gitlab/schemas/group_variables.json b/tap_gitlab/schemas/group_variables.json
@@ -0,0 +1,26 @@
+{
+    "type": "object",
+    "properties": {
+        "group_id": {
+            "type": ["null", "integer"]
+        },
+        "variable_type": {
+            "type": ["null", "string"]
+        },
+        "key": {
+            "type": ["null", "string"]
+        },
+        "value": {
+            "type": ["null", "string"]
+        },
+        "protected": {
+            "type": ["null", "boolean"]
+        },
+        "masked": {
+            "type": ["null", "boolean"]
+        },
+        "environment_scope": {
+            "type": ["null", "string"]
+        }
+    }
+}