Skip to content

Fix having a submodule as a primary source#133

Merged
marmarek merged 6 commits into
QubesOS:mainfrom
marmarek:main-src-submodule
Jul 18, 2024
Merged

Fix having a submodule as a primary source#133
marmarek merged 6 commits into
QubesOS:mainfrom
marmarek:main-src-submodule

Conversation

@marmarek

@marmarek marmarek commented Jul 8, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Do not try to interpret Source0 tag to build automatic tarball name, if
it isn't going to be created. This is especially relevant when Source0
points at a submodule archive (created via "modules" entry in
.qubesbuilder).

TODO:

  • add a test (with a package actually having Source0: @something@)
  • verify if Debian and/or Arch doesn't need similar change

Fixes QubesOS/qubes-issues#9088

@marmarek
Fix having a submodule as a primary source
cc14dbb 
Do not try to interpret Source0 tag to build automatic tarball name, if
it isn't going to be created. This is especially relevant when Source0
points at a submodule archive (created via "modules" entry in
.qubesbuilder).

Fixes QubesOS/qubes-issues#9088
@marmarek

marmarek commented Jul 10, 2024

Copy link
Copy Markdown
Member Author

@fepitre see if you are okay with this approach as a solution for QubesOS/qubes-issues#9088. If so, I'll fill missing items from the TODO above.

@fepitre

fepitre commented Jul 11, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Do you have an example for this case or this is for some radical change on providing sources?

@marmarek

marmarek commented Jul 11, 2024

Copy link
Copy Markdown
Member Author

yes: QubesOS/qubes-vmm-xen#185

@fepitre

fepitre commented Jul 11, 2024

Copy link
Copy Markdown
Member

Ok I see, so we need to improve the phase below create archive or what ever is needed to trigger archive creation if we provide necessary information from .qubesbuilder

@marmarek

marmarek commented Jul 11, 2024 •
edited
Loading

Copy link
Copy Markdown
Member Author

Here I used already existing functionality of modules entry in .qubesbuilder instead of standard create archive. The issue is, some parts of create archive were still active (especially building archive name), which failed when I used @xen@ placeholder.

Alternative would be yet another type for files section - like when we have url now, maybe add git-url, and then have also commit-id/tag/branch + gpg key to verify (for tag or branch - commit id explicitly doesn't require anything extra). But since we support submodules, I thought it will be easier this way.

@fepitre

fepitre commented Jul 11, 2024

Copy link
Copy Markdown
Member

Alternative would be yet another type for files section - like when we have url now, maybe add git-url, and then have also commit-id/tag/branch + gpg key to verify (for tag or branch - commit id explicitly doesn't require anything extra). But since we support submodules, I thought it will be easier this way.

I like the idea, if entry is git-url instead of url, there we can add commit/branch/tag in the process, in that case we can keep other fields like pubkeys.

marmarek added 4 commits July 13, 2024 03:29
@marmarek
Refactor fetching normal files into a function
656fda6 
No functional change.
@marmarek
Factor out get_archive_name function
97955ba 
Right now it's trivial, but it will get extended in further commits.
@marmarek
Allow git branch starting with a number
be11c38 
Branch name starting with a number is a valid thing, and it's used by
some projects for stable release branches. Additionally, some projects
tag versions with any prefix before the actual version, and the script
otherwise work with tag name as a branch name.
@marmarek
Extend get-and-verify-source.py script
00ab895 
Add a bunch new options:
--git-commits - fetch explicit, pre-verified, commit id
--shallow-clone - skip fetching git history to speed up fetching
--trust-all-keys - import and trust all keys from the keys-dir; useful
with dedicated keys dir

Add also tests for the new functionality.
@marmarek
marmarek force-pushed the main-src-submodule branch from f2abbca to 2dede4c Compare July 13, 2024 01:29
@marmarek
Add support for 'git-url' in 'files' section of .qubesbuilder
c4ab9f6 
This allows fetching upstream sources from a git repository directly.
Not all projects produce release tarballs, some use only
github-generated tarballs (which aren't always reproducible).
Additionally, as xz-utils story shows, manual release tarballs is
additional attack vector in the supply chain.

Right now support two modes:
1. Specific commit id, for example:
```
files:
- git-url: https://github.com/some/repository
  commit-id: (full sha of the commit)
  git-basename: something-@Version@
```

2. Signed tag, for example:
```
files:
- git-url: https://github.com/some/repository
  tag: v@VERSION@
  pubkeys:
  - pubkey.asc
```

Fixes QubesOS/qubes-issues#9088
@marmarek
marmarek force-pushed the main-src-submodule branch from 2dede4c to c4ab9f6 Compare July 13, 2024 01:35
@codecov

codecov Bot commented Jul 13, 2024 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 70.77922% with 45 lines in your changes missing coverage. Please review.

Project coverage is 77.58%. Comparing base (dbf9a17) to head (c4ab9f6).
Report is 1 commits behind head on main.

Files Patch % Lines
qubesbuilder/plugins/fetch/__init__.py 63.63% 36 Missing ⚠️
...der/plugins/fetch/scripts/get-and-verify-source.py 84.61% 4 Missing ⚠️
qubesbuilder/plugins/source_rpm/__init__.py 60.00% 4 Missing ⚠️
qubesbuilder/common.py 93.75% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #133      +/-   ##
==========================================
+ Coverage   75.94%   77.58%   +1.64%     
==========================================
  Files          46       46              
  Lines        5059     5144      +85     
==========================================
+ Hits         3842     3991     +149     
+ Misses       1217     1153      -64

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marmarek
marmarek marked this pull request as ready for review July 18, 2024 14:39
@marmarek
marmarek requested a review from fepitre July 18, 2024 14:40
fepitre
fepitre reviewed Jul 18, 2024
View reviewed changes
Comment thread tests/test_cli.py
/ "distfiles/linux-gbulb/gbulb-0.6.3.tar.gz"
).exists()
# verify files layout inside
subprocess.run(

@fepitre fepitre Jul 18, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you verify? Is this looking at logs only?

@fepitre fepitre Jul 18, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I missed the line that you look for README.rst

fepitre
fepitre approved these changes Jul 18, 2024
View reviewed changes

@fepitre fepitre left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Approve HEAD being at commit c4ab9f6
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEn6ZLkvlecGvyjiymSEAQtc3FduIFAmaZMBQACgkQSEAQtc3F
duKjyw//UQ2Vu5kqKqRTQXu5yzzw47eUbwMQUpQLqLQW8iH7J1aBwYqasESnz32K
WjVFfk3n42dAfoRjV3LBaWSZEoZ2LF+htxWkE7hSUsugLhtHcDW7aFv4qnwWakcg
F3iGVvoRtIdt6+QLIr6sAhNc6eDJda4odSM6Zh0Smt5+pXqE94OYI36jxxiIbJZl
AFzCirjhagyWWed9borPLVruJgBAnaE1S8OdcKbzTD3BHCeZs2NM2EqVG4A9/Myk
qd87HiZFg4scKr5xEfGok2SsUF0jeNyHXthiRDmtaorvB8UOPyLTkDInuxGXu00D
wyy6Z1gIWqBQGYCuNFSLli/UP+CIAgmANjqDIl9Zn1jlmBNDfKDXZ+R5N0dRXDCk
SDHwohnmbAKrAIzPWzu/YO6SMr1Xr45/1ZB7PWVDd0y7C13jmRxniOjRKC3yw8PA
2wz2zX3N9NDSXOg1qW1DnDd0pHt36PIlOAS//korlacgjz4ARoENM4Y/sYUsXLYz
hMOcLk4EATGkdmOcR8nrDHZ1t5utOwedr5GEYGSM7xSq82V/8ygEKfP3aBG0ISQ9
IuKzsJS6Uw1p9Oog/tOQuLf7mTKjYq+nBhXLLcHX2LSLthwM8FbD19NXRzDPvBBE
yp9ifO8r5i2qd03sj40i8+ND4GM8oPFVUooLQwaYpHlyx1kCI0M=
=XBtf
-----END PGP SIGNATURE-----

@marmarek
marmarek merged commit c382ce5 into QubesOS:main Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@fepitre fepitre fepitre approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

builderv2: support fetching sources from git (and making tarball out of it for packaging purposes)

2 participants

@marmarek @fepitre