-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[vcpkg-ci] Add test port for cmake user projects #19034
Conversation
x64-linux does work locally. Also when merging origin/master. |
linux: cmake 3.4 can't find libidn.so.11. |
1595e08
to
3066d11
Compare
Ready for review, but leaving draft state until #18898 is merged. |
@BillyONeal Maybe you are already interested in looking at this PR. I'm loosing patience with waiting for #18898. |
I don't believe doing this as a test port is the appropriate thing, perhaps an e2e test ( https://github.com/microsoft/vcpkg-tool/tree/main/azure-pipelines/end-to-end-tests-dir )? Ports are supposed to always get the version of cmake vcpkg provides (or later), not old ones. |
This is clear and misses the point of this PR. This PR isn't to test building a port with an older version: It is to simulate a user project environment. This "port" is just a hook into the automated testing triggered by relevant changes to scripts and ports. For now, it triggers with changes to selected popular ports (catching incompatible changes to wrappers). It should also trigger from changes to vcpkg.cmake scripts if that were part of the port ABI but I don't think this is the case. It might be related to e2e tests by its "nature", but IIUC these tests are triggered from changing the tool. This doesn't seem the adequate trigger for testing cmake compatibility bugs. The cmake 3.1 (actual 3.4) promise is very likely to be broken by changes to toolchain scripts and wrappers (CMake code), not by changes to the tools (C++ code). |
While reviewing my open PRs which need work I stumbled over the merge conflict for #19120. I realized that the changes already merged to |
@BillyONeal Can you please give directions how to move this forward? Your last post indicated a misunderstanding of what this port is adding to CI, and I tried to rectify this. |
The bits in the "scripts" tree are logically part of the tool (and the scripts tree will be moving there too very soon). I wasn't trying to say that this tests
I don't know if we're waiting on anything from you specifically right now. |
IMO the test is for validating changed ports so I'm actually confused by the suggestion to move it to e2e tests. Perhaps it might be a post-build check instead. In any case, it needs to run before merging changes to the vcpkg repo when someone modifies a port.
I admit that the highlighting of errors in AZP/Github could be more explicit than just reporting a regression for port cmake-user, in order to reduce confusion. |
Okay, in this test port are tests looking at other ports (feature |
Done (trivial). |
Seems all request changes have been applied. Could you please help take a look again if this is ready to merge now? @BillyONeal |
@BillyONeal and @ras0219-msft Could you please help review this PR again? Since all requests seem to be applied. |
@BillyONeal CI isn't uploading artifacts for osx, also not in microsoft.vcpkg.ci:
|
Yeah the credentials expired Friday evening :(. #20512 is the fix. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Ping |
All checks green again. |
@vicroms, @BillyONeal Could you please help review and merge this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This generally looks good and should function, however I note there is one significant drawback:
By testing every library as a single node, we require restoring binary packages and performing a configure for every library when any library changes. Before this can scale out to 100s of libraries, we will need to change the approach to produce individual test ports per package.
As a suggestion for how this could be done: the comment fields for CMake usage information could be distributed into the producing port. During CI, a script scans for these fields and generates test ports for each cmake usage into an overlay folder. Then, we rely on the normal binary caching / ci procedures to efficiently manage which tests to run.
While this port is indeed testing libraries "as a single node", it probably won't depend on more than a limited set of ports soon. But it can be exploited locally for pre-PR test on other ports. When tests are scripted in CI only, such exploitation is much harder. A future replacement must be implemented in vcpkg. |
What does your PR fix?
This PR adds a port which will test configuring, building and installing user projects with the vcpkg cmake toolchain with regard to the following areas:
find_package()
success and linking<PKG>_LIBRARIES
for selected ports (via dependencies in vcpkg.json).Ideally, this port would always rebuild in CI when toolchain files change.
This PR complements [vcpkg] Fix toolchain compatibility with cmake < 3.15 #18898. Only the changes to
scripts/test_ports/cmake_user
(+baseline) are new work.Which triplets are supported/not supported? Have you updated the CI baseline?
all triplets, but minimum version test only for x64 windows and osx.
CI baseline: Skipping x64-linux until package
libidn11
is provisioned in order to run cmake 3.4.Does your PR follow the maintainer guide?
yes
If you have added/updated a port: Have you run
./vcpkg x-add-version --all
and committed the result?Test port, not registered.