Mirror mojo fails due cache files being too long

[rbioteau]

When mirroring a target platform containing the following repository

<plugin>
            <groupId>org.eclipse.tycho.extras</groupId>
            <artifactId>tycho-p2-extras-plugin</artifactId>
            <version>4.0.10</version>
            <executions>
              <execution>
                <phase>prepare-package</phase>
                <goals>
                  <goal>mirror</goal>
                </goals>
              </execution>
            </executions>
            <configuration>
              <source>
                <repository>
                  <url>https://groovy.jfrog.io/artifactory/plugins-release/e4.33</url>
                  <layout>p2</layout>
                </repository>
              </source>
              <ius>
                <iu>
                  <id>org.codehaus.groovy.compilerless.feature.feature.group</id>
                  <version>5.5.0.v202409302028-e2409</version>
                </iu>
                <iu>
                  <id>org.codehaus.groovy.m2eclipse.feature.feature.group</id>
                  <version>5.5.0.v202409302028-e2409</version>
                </iu>
                <iu>
                  <id>org.codehaus.groovy30.feature.feature.group</id>
                  <version>5.5.0.v202409302028-e2409</version>
                </iu>
                <iu>
                  <id>org.codehaus.groovy.eclipse.feature.feature.group</id>
                  <version>5.5.0.v202409302028-e2409</version>
                </iu>
              </ius>
            </configuration>
          </plugin>

The mojo fails with the following error:

download from https://groovy.jfrog.io/artifactory/plugins-release-local/org/codehaus/groovy/groovy-eclipse-integration/5.5.0/e4.33/plugins/org.codehaus.groovy_4.0.23.v202409302028-e2409.jar failed]: /home/romain/.m2/repository/.cache/tycho/https/storage.googleapis.com/art-groovy-filestore-1478119448/filestore/fd/fd701dc97c13a0248cdd59f861529c6f40f84670/X-Artifactory-artifactPath=org%2Fcodehaus%2Fgroovy%2Fgroovy-eclipse-integration%2F5.5.0%2Fe4.33%2Fplugins%2Forg.codehaus.groovy_4.0.23.v202409302028-e2409.jar/X-Artifactory-originPackageType=maven/X-Artifactory-originProjectKey=default/X-Artifactory-originRepoType=local/X-Artifactory-originRepositoryKey=plugins-release-local/X-Artifactory-packageType=maven/X-Artifactory-projectKey=default/X-Artifactory-repoType=local/X-Artifactory-repositoryKey=plugins-release-local/X-Artifactory-username=anonymous/X-Goog-Algorithm=GOOG4-RSA-SHA256/X-Goog-Credential=gro7976160665383240%40aol-prod.iam.gserviceaccount.com%2F20241129%2Fauto%2Fstorage%2Fgoog4_request/X-Goog-Date=20241129T100207Z/X-Goog-Expires=60/X-Goog-SignedHeaders=host/response-content-disposition=attachment%3Bfilename%3D%22org.codehaus.groovy_4.0.23.v202409302028-e2409.jar%22/response-content-type=application%2Fjava-archive/x-jf-traceId=3b1a4ee0e941c91a/X-Goog-Signature=1f19412cfab85789a5a9df9dc1890de7536a2116657a9d9087d4517ec891b9ef0b62a43feea9e547c1eaeae448a268b14929ed7de30cd6fe565cdecafc36e5bef3469bc5b889ad30c8e5f4794f041b51a687f2e9899e311fd98134af6bb41098ffd8d061b0bab11eb87b67185bc02289051f9c187784ac07f5f87c89a7478663892f91815213ac7914b95a5276c54ef6cb3b16c240d6e9560e76a9dd86fb3ea07139d398e308fcc684d31d650e8e4111465722c8694e920224f76d00ebaf1bbe7068b603836567df3b95d39f629001657aaa1212ede9e9606f0c8e6451c7b2222e7d74454a68ef8314dbb65399c766a9400e7164b1b171445d36ea464a3bc5ca.headers (Name is too long) -> [Help 1]

[rbioteau]

I manage to workaround the issue patching the SharedHttpCacheStorage.

diff --git a/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport/SharedHttpCacheStorage.java b/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport/SharedHttpCacheStorage.java
index b39b3c5..70ffa9b 100644
--- a/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport/SharedHttpCacheStorage.java
+++ b/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport/SharedHttpCacheStorage.java
@@ -251,17 +251,30 @@
 				}
 				updateHeader(response, code);
 				if (isRedirected(code)) {
-					File cachedFile = SharedHttpCacheStorage.this.getCacheEntry(getRedirect(uri), logger)
-							.getCacheFile(transportFactory);
-					// https://github.com/eclipse-tycho/tycho/issues/2938
-					// Redirect may change extension. P2's SimpleMetadataRepositoryFactory relies on
-					// accurate file extension to be cached.
-					// Copying file to accommodate original request and its file extension.
-					// Once https://github.com/eclipse-equinox/p2/issues/355 is fixed, cachedFile
-					// may be returned directly without copying.
-					response.close(); // early close before doing unrelated file I/O
-					FileUtils.copyFile(cachedFile, file);
-					return file;
+					try {
+						File cachedFile = SharedHttpCacheStorage.this.getCacheEntry(getRedirect(uri), logger)
+								.getCacheFile(transportFactory);
+						// https://github.com/eclipse-tycho/tycho/issues/2938
+						// Redirect may change extension. P2's SimpleMetadataRepositoryFactory relies on
+						// accurate file extension to be cached.
+						// Copying file to accommodate original request and its file extension.
+						// Once https://github.com/eclipse-equinox/p2/issues/355 is fixed, cachedFile
+						// may be returned directly without copying.
+						response.close(); // early close before doing unrelated file I/O
+						FileUtils.copyFile(cachedFile, file);
+						return file;
+					}catch (IOException e) {
+						logger.warn("Failed to cache file: " + e.getMessage());
+
+						HttpTransport redirectTransport = transportFactory.createTransport(getRedirect(uri));
+						redirectTransport.get(r -> {
+							try (OutputStream os = new BufferedOutputStream(new FileOutputStream(file))) {
+								r.transferTo(os);
+							}
+							return null;
+						});
+						return file;
+					}
 				}
 				if (exits) {
 					FileUtils.forceDelete(file);

[kysmith-csg]

@rbioteau could you submit this as a pull request? We are facing the same issue (also with jfrog).

[TIBCOrkrajews]

Caching using redirected URLs to cloud/CDN providers as the key is unlikely to function as a cache because the URL is likely to contain a token with a very short lifespan. Repeated requests to the same original URL will yield different redirected URLs and there will never be a cache hit for more than a few minutes.

[kysmith-csg]

@rbioteau we really need this fix, are you able to make a PR? Otherwise I will, but I don't want to steal your work.

[rbioteau]

@rbioteau we really need this fix, are you able to make a PR? Otherwise I will, but I don't want to steal your work.

Hi, no I didn't make a PR and I'm not confident enough to suggest one as I don't understanding all the ins and outs of this cache mechanism.
The patch I posted was just to help others working around the issue but it certainly not a viable solution.

Feel free to work on this topic if you feel more confident.

[laeubi]

@TIBCOrkrajews P2 is definitely not designed for usage with dynamic redirects, in this case I would recommend do use some kind of proxy that allows for stable URLs as otherwise you will get poor performance. P2 itself already has the concept of mirror servers for artifacts, so one would better use that then if it is really required.

When working on a solution one might distinguish permanent from temporal redirects and do not cache temporal ones, but this will result in the offline functionality to render completely useless, so one should take this into consideration.

[kysmith-csg]

@laeubi the way our Artifactory is set up (thanks to enterprise IT), the redirection isn't even consistent. Sometimes it redirects, but sometimes it doesn't.

What I was thinking as a solution is to cache the artifact under the original URL for 302 Found, and cache it under the redirected URL for 301 Moved Permanently (the current behavior for all redirections). This seems to align with the HTTP standard:

The 302 (Found) status code indicates that the target resource
resides temporarily under a different URI. Since the redirection
might be altered on occasion, the client ought to continue to use the
effective request URI for future requests.

Compared to 301 Moved Permanently:

The 301 (Moved Permanently) status code indicates that the target
resource has been assigned a new permanent URI and any future
references to this resource ought to use one of the enclosed URIs.

---

When working on a solution one might distinguish permanent from temporal redirects and do not cache temporal ones, but this will result in the offline functionality to render completely useless, so one should take this into consideration.

I don't think I would implement this without making some kind of system property to enable it (unless you think it's fine). It seems like a much easier solution than what I'm proposing :)

[laeubi]

Currently the cache just tries to "replay" but in case of redirects we could/should probably improve this yes it was not written with such things in mind and I doubt there is even any tests yet.