Fix -with-sources not fetching differently named source packages

[schoenherrg]

Fixes #1403

Description of the Change

This fixes a bug where aptly doesn't behave as specified in the documentation.

The core of the change is to just add an option to Filter to indicate that source packages need to be included and, if that option is set, add the missing packages to the result.

--- a/deb/list.go
+++ b/deb/list.go
@@ -524,6 +537,37 @@ func (l *PackageList) Filter(options FilterOptions) (*PackageList, error) {
  for _, query := range options.Queries {
          _ = result.Append(query.Query(l))
  }
+ // The above loop already finds source packages that are named equal to their binary package, but we still need
+ // to account for those that are named differently.
+ if options.WithSources {
+         sourceQueries := make([]PackageQuery, 0)
+         for _, pkg := range result.packages {
+                 // Find sourceName, sourceVersion
+                 // ...
+                 sourceQueries = append(sourceQueries, &DependencyQuery{Dependency{
+                         Pkg:          sourceName,
+                         Version:      sourceVersion,
+                         Relation:     VersionEqual,
+                         Architecture: ArchitectureSource,
+                 }})
+         }
+         for _, query := range sourceQueries {
+                 _ = result.Append(query.Query(l))
+         }
+ }

Note that I also refactored Filter to take a struct of options, because I didn't want to add yet another argument to the already very long argument list.

--- a/deb/list.go
+++ b/deb/list.go

-// Filter filters package index by specified queries (ORed together), possibly pulling dependencies
-func (l *PackageList) Filter(queries []PackageQuery, withDependencies bool, source *PackageList, dependencyOptions int, architecturesList []string) (*PackageList, error) {
-   return l.FilterWithProgress(queries, withDependencies, source, dependencyOptions, architecturesList, nil)
+// FilterOptions specifies options for Filter()
+type FilterOptions struct {
+ Queries           []PackageQuery
+ WithDependencies  bool
+ Source            *PackageList
+ DependencyOptions int
+ Architectures     []string
+ Progress          aptly.Progress // set to non-nil to report progress
 }

-// FilterWithProgress filters package index by specified queries (ORed together), possibly pulling dependencies and displays progress
-func (l *PackageList) FilterWithProgress(queries []PackageQuery, withDependencies bool, source *PackageList, dependencyOptions int, architecturesList []string, progress aptly.Progress) (*PackageList, error) {
+// Filter filters package index by specified queries (ORed together), possibly pulling dependencies
+func (l *PackageList) Filter(options FilterOptions) (*PackageList, error) {

A call to Filter now looks like this (you can leave out arguments that have the default value):

-   repo.packageList, err = repo.packageList.FilterWithProgress([]PackageQuery{filterQuery}, repo.FilterWithDeps, emptyList, dependencyOptions, repo.Architectures, progress)
+ repo.packageList, err = repo.packageList.Filter(FilterOptions{
+         Queries:           []PackageQuery{filterQuery},
+         WithDependencies:  repo.FilterWithDeps,
+         Source:            emptyList,
+         DependencyOptions: dependencyOptions,
+         Architectures:     repo.Architectures,
+         Progress:          progress,
+ })

This change is independent of the bugfix though and I can revert it if you don't like it.

Checklist

• unit-test added (if change is algorithm)
• functional test added/updated (if change is functional)
• author name in AUTHORS

[schoenherrg]

For the test PublishSnapshot41Test, I am not sure what to do. The mirror that this test is using (http://repo.aptly.info/system-tests/security.debian.org/debian-security) seems to not contain the required source packages. This was previously not an issue because not Aptly was just ignoring those source packages even though the flag -with-sources is specified. Should I change the test and remove the -with-sources flag?

    ERROR: unable to update: download errors:
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/u/util-linux/util-linux_2.33.1-0.1+deb10u1.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/u/util-linux/util-linux_2.33.1-0.1+deb10u1.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/u/util-linux/util-linux_2.33.1.orig.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/g/glibc/glibc_2.28-10+deb10u2.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/g/glibc/glibc_2.28-10+deb10u2.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/g/glibc/glibc_2.28.orig.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/n/ncurses/ncurses_6.1+20181013-2+deb10u5.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/n/ncurses/ncurses_6.1+20181013-2+deb10u5.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/n/ncurses/ncurses_6.1+20181013.orig.tar.gz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/z/zlib/zlib_1.2.11.dfsg-1+deb10u2.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/z/zlib/zlib_1.2.11.dfsg-1+deb10u2.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/z/zlib/zlib_1.2.11.dfsg.orig.tar.gz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/x/xz-utils/xz-utils_5.2.4-1+deb10u1.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/x/xz-utils/xz-utils_5.2.4-1+deb10u1.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/x/xz-utils/xz-utils_5.2.4.orig.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/e/e2fsprogs/e2fsprogs_1.44.5-1+deb10u2.debian.tar.xz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/e/e2fsprogs/e2fsprogs_1.44.5-1+deb10u2.dsc
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/e/e2fsprogs/e2fsprogs_1.44.5.orig.tar.gz
      HTTP code 404 while fetching http://repo.aptly.info/system-tests/security.debian.org/debian-security/pool/updates/main/e/e2fsprogs/e2fsprogs_1.44.5.orig.tar.gz.asc

[5hir0kur0]

@neolynx this PR is actually from me. I was told to contribute this with an @siemens.com email/account because I'm doing this for Siemens and they want more visibility for the open source contributions that they paid for.

[neolynx]

the test is failing due to missing files in the S3 mirror for system tests.

I will add those..

[neolynx]

additional files added, test are green now :)

codecov/patch — 90.59% of diff hit (target 74.81%)

nice job 👍

[5hir0kur0]

additional files added, test are green now :)

codecov/patch — 90.59% of diff hit (target 74.81%)

nice job 👍

Thank you 😊

[neolynx]

@schoenherrg could you provide me access to the PR ?

I wanted to push an update on the system test, checking for the additional source pkgs with different name explicitely...