Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downloader: Skip zero size artifacts #8116

Closed
timo-HERE opened this issue Jan 15, 2024 · 7 comments
Closed

Downloader: Skip zero size artifacts #8116

timo-HERE opened this issue Jan 15, 2024 · 7 comments
Labels
downloader About the downloader tool enhancement Issues that are considered to be enhancements good first issue An easy issue for new contributors help wanted An issue where third-party help is wanted on

Comments

@timo-HERE
Copy link

Sometimes ORT comes across with zero size source artifacts. Those obviously are errors that should be fixed/deleted from the repository, but I think ORT also should gracefully ignore those files instead of trying to unpack 0 byte files.

ERROR org.ossreviewtoolkit.downloader.Downloader - Could not unpack source artifact '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar': IOException: Unable to unpack '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar'. This file is not a supported archive type.
Suppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as ZIP failed.
    Caused by: IOException: Error on ZipFile /tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar
        Caused by: ZipException: Archive is not a ZIP archiveSuppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as SEVENZIP failed.
    Caused by: EOFException: nullSuppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as TAR_BZIP2 failed.
    Caused by: IOException: Stream is not in the BZip2 formatSuppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as TAR_GZIP failed.
    Caused by: IOException: Input is not in the .gz formatSuppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as TAR_XZ failed.
    Caused by: EOFException: nullSuppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as TAR failed.
    Caused by: IOException: Unsupported archive type or empty archive.Suppressed: IOException: Unpacking '/tmp/ort-Downloader2786093702508072637/xmlpull-1.1.3.1-sources.jar' as DEB failed.
    Caused by: IOException: Failed to read header. Occurred at byte: 0
09:50:35.435 [main] ERROR org.ossreviewtoolkit.downloader.Downloader - Could not unpack source artifact '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar': IOException: Unable to unpack '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar'. This file is not a supported archive type.
Suppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as ZIP failed.
    Caused by: IOException: Error on ZipFile /tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar
        Caused by: ZipException: Archive is not a ZIP archiveSuppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as SEVENZIP failed.
    Caused by: EOFException: nullSuppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_BZIP2 failed.
    Caused by: IOException: Stream is not in the BZip2 formatSuppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_GZIP failed.
    Caused by: IOException: Input is not in the .gz formatSuppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_XZ failed.
    Caused by: EOFException: nullSuppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR failed.
    Caused by: IOException: Unsupported archive type or empty archive.Suppressed: IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as DEB failed.
    Caused by: IOException: Failed to read header. Occurred at byte: 0
Exception in thread "main" org.ossreviewtoolkit.downloader.DownloadException: java.io.IOException: Unable to unpack '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar'. This file is not a supported archive type.
	at org.ossreviewtoolkit.downloader.Downloader.downloadSourceArtifact(Downloader.kt:377)
	at org.ossreviewtoolkit.downloader.Downloader.downloadSourceArtifact$default(Downloader.kt:303)
	at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.download(ProvenanceDownloader.kt:68)
	at org.ossreviewtoolkit.scanner.Scanner.downloadRecursively(Scanner.kt:745)
	at org.ossreviewtoolkit.scanner.Scanner.createMissingArchives(Scanner.kt:721)
	at org.ossreviewtoolkit.scanner.Scanner.scan(Scanner.kt:181)
	at org.ossreviewtoolkit.scanner.Scanner$scan$3.invokeSuspend(Scanner.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:280)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:227)
	at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:140)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:306)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:319)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:40)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:458)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:455)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:475)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:482)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:85)
Caused by: java.io.IOException: Unable to unpack '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar'. This file is not a supported archive type.
	at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:119)
	at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes$default(ArchiveUtils.kt:105)
	at org.ossreviewtoolkit.downloader.Downloader.downloadSourceArtifact(Downloader.kt:369)
	... 24 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as ZIP failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.IOException: Error on ZipFile /tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:655)
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:552)
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:531)
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:458)
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:[446](https://main.gitlab.in.here.com/oss/oss-review-toolkit/ort-gitlab-ci/-/jobs/87866910#L446))
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackZip(ArchiveUtils.kt:171)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:86)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Caused by: java.util.zip.ZipException: Archive is not a ZIP archive
		at org.apache.commons.compress.archivers.zip.ZipFile.positionAtEndOfCentralDirectoryRecord(ZipFile.java:1146)
		at org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:1034)
		at org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:1009)
		at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:648)
		... 33 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader781122818322[462](https://main.gitlab.in.here.com/oss/oss-review-toolkit/ort-gitlab-ci/-/jobs/87866910#L462)8765/xmlpull-1.1.3.1-sources.jar' as SEVENZIP failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.EOFException
		at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:278)
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.readFully(SevenZFile.java:1299)
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.readHeaders(SevenZFile.java:1335)
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:[481](https://main.gitlab.in.here.com/oss/oss-review-toolkit/ort-gitlab-ci/-/jobs/87866910#L481))
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:332)
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:345)
		at org.apache.commons.compress.archivers.sevenz.SevenZFile.<init>(SevenZFile.java:291)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack7Zip(ArchiveUtils.kt:129)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:85)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_BZIP2 failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.IOException: Stream is not in the BZip2 format
		at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.init(BZip2CompressorInputStream.java:567)
		at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:293)
		at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:271)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:89)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_GZIP failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.IOException: Input is not in the .gz format
		at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:242)
		at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:189)
		at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:153)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:90)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR_XZ failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.EOFException
		at java.base/java.io.DataInputStream.readFully(DataInputStream.java:203)
		at java.base/java.io.DataInputStream.readFully(DataInputStream.java:172)
		at org.tukaani.xz.SingleXZInputStream.readStreamHeader(Unknown Source)
		at org.tukaani.xz.SingleXZInputStream.<init>(Unknown Source)
		at org.tukaani.xz.SingleXZInputStream.<init>(Unknown Source)
		at org.tukaani.xz.SingleXZInputStream.<init>(Unknown Source)
		at org.apache.commons.compress.compressors.xz.XZCompressorInputStream.<init>(XZCompressorInputStream.java:134)
		at org.apache.commons.compress.compressors.xz.XZCompressorInputStream.<init>(XZCompressorInputStream.java:102)
		at org.apache.commons.compress.compressors.xz.XZCompressorInputStream.<init>(XZCompressorInputStream.java:79)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:91)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as TAR failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.IOException: Unsupported archive type or empty archive.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:305)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTar(ArchiveUtils.kt:259)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:88)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
	Suppressed: java.io.IOException: Unpacking '/tmp/ort-Downloader7811228183224628765/xmlpull-1.1.3.1-sources.jar' as DEB failed.
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:117)
		... 26 more
	Caused by: java.io.IOException: Failed to read header. Occurred at byte: 0
		at org.apache.commons.compress.archivers.ar.ArArchiveInputStream.getNextArEntry(ArArchiveInputStream.java:264)
		at org.apache.commons.compress.archivers.ar.ArArchiveInputStream.getNextEntry(ArArchiveInputStream.java:350)
		at org.apache.commons.compress.archivers.ar.ArArchiveInputStream.getNextEntry(ArArchiveInputStream.java:36)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:278)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackDeb(ArchiveUtils.kt:226)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpack(ArchiveUtils.kt:93)
		at org.ossreviewtoolkit.utils.common.ArchiveUtilsKt.unpackTryAllTypes(ArchiveUtils.kt:113)
		... 26 more
@sschuberth
Copy link
Member

sschuberth commented Jan 15, 2024 •
edited
Loading

Those obviously are errors that should be fixed/deleted from the repository, but I think ORT also should gracefully ignore those files instead of trying to unpack 0 byte files.

How would a user become aware of these errors in order to fix them if ORT would "gracefully ignore those files"? Or do you mean that an error should be logged, but no exception should be thrown?

But actually, I've not seen this before myself, so I'm not sure how worthwhile a special handling for 0-byte files would be.

@sschuberth sschuberth added enhancement Issues that are considered to be enhancements downloader About the downloader tool labels Jan 15, 2024
@timo-HERE
Copy link
Author

This can be shown as an issue, but the scan should continue.
Current behaviour is that this exception stops the execution.

@sschuberth sschuberth added help wanted An issue where third-party help is wanted on good first issue An easy issue for new contributors labels Jan 25, 2024
@timo-HERE
Copy link
Author

timo-HERE commented Feb 7, 2024 •
edited
Loading

Most likely the root cause was an issue with osgeo repository (repo.osgeo.org) which happened last year.

Discussion: https://www.mail-archive.com/[email protected]/msg20339.html
Issue ticket: https://trac.osgeo.org/osgeo/ticket/2978

@fviernau
Copy link
Member

I've also just came across this. What surprises me, is that the logs look as if the exception is not caught.
But, looking at the code - it should be caught. So, maybe the exception in the log is not the root cause of the crash ?

See:

ort/scanner/src/main/kotlin/Scanner.kt

Lines 705 to 719 in bf5661b

try {
dir = provenanceDownloader.downloadRecursively(nestedProvenance)
archiver.archive(dir, nestedProvenance.root)
} catch (e: IOException) {
controller.addIssue(
pkg.id,
Issue(
source = "Scanner",
message = "Could not create file archive for " +
"'${pkg.id.toCoordinates()}': ${e.collectMessages()}"
)
)
} finally {
dir?.safeDeleteRecursively(force = true)
}

@sschuberth
Copy link
Member

I guess #8303 helps?

@sschuberth
Copy link
Member

sschuberth commented Feb 19, 2024 •
edited
Loading

@timo-HERE could you please check whether the issue is resolved for you with the latest main? I.e. 0-byte artifacts are still not skipped explicitly, but coming across them should now create an issue entry instead of throwing an exception.

fviernau added a commit that referenced this issue Feb 19, 2024
@fviernau
fix(gradle-inspector): Ignore zero by size artifact archive files
6849f66 
Gradle artifacts are archive files. A file with zero bytes size never
is a valid archive, so fallback to `RemoteArtifact.EMPTY` in that case.

This fixes one possible rot cause of #8116.

Signed-off-by: Frank Viernau <[email protected]>
fviernau added a commit that referenced this issue Feb 19, 2024
@fviernau
fix(gradle-inspector): Ignore zero by size artifact archive files
769f34d 
Maven artifacts are archive files. A file with zero bytes size never
is a valid archive, so fallback to `RemoteArtifact.EMPTY` in that case.

This fixes one possible root cause of #8116.

Signed-off-by: Frank Viernau <[email protected]>
sschuberth pushed a commit that referenced this issue Feb 19, 2024
@fviernau @sschuberth
fix(gradle-inspector): Ignore zero by size artifact archive files
25c9790 
Maven artifacts are archive files. A file with zero bytes size never
is a valid archive, so fallback to `RemoteArtifact.EMPTY` in that case.

This fixes one possible root cause of #8116.

Signed-off-by: Frank Viernau <[email protected]>
@sschuberth
Copy link
Member

Feel free to reopen if the issue persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
downloader About the downloader tool enhancement Issues that are considered to be enhancements good first issue An easy issue for new contributors help wanted An issue where third-party help is wanted on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sschuberth @fviernau @timo-HERE