vcs2l fails with scenario_simulator_v2

[mitsudome-r]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.

Description

After #6644 was merged, scneario_simulator_v2 fails to build with the instructions in https://autowarefoundation.github.io/autoware-documentation/main/demos/scenario-simulation/scenario-simulator/installation/.

It will fail during rosdep install with the following error message:

executing command [apt-get install -y ros-humble-ament-cmake-vendor-package]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3-vcs2l : Conflicts: python3-vcstool but 0.3.0-1 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
ERROR: the following rosdeps failed to install
  apt: command [apt-get install -y ros-humble-ament-cmake-vendor-package] failed

Cause of the problem

The error occurs because zmqpp_vendor package depends on ament_cmake_vendor_packages, and ament_cmake_vendor_package has dependency on python3-vcstool which conflicts with vcs2l.

Possible solutions

• Approach 1: update ament_cmake_vendor_packages to use vcs2l instead
• Approach 2: split zmqpp_vendor out from scenario_simulator and release it on ROS buildfarm.

[mitsudome-r]

I would propose to revert the commit for now until we have a proper fix made.

[xmfcx]

• Approach 1 implementation: Migrate package dependency from vcstool to vcs2l. ament/ament_cmake#601

I guess then for now, we can focus on this being on rolling and lyrical so that it does not affect the currently active LTS distributions.

This option might take forever to implement.

Approach 2:

We should have TIER IV put the https://github.com/tier4/scenario_simulator_v2/tree/master/external/zmqpp_vendor or https://github.com/tier4/zmqpp_vendor package to the ROS build farm.

---

Until then, I will continue without vcs2l for now 😢

[xmfcx]

It turns out this package is already in the build farm from TIER IV.

• https://github.com/tier4/zmqpp_vendor
• https://index.ros.org/p/zmqpp_vendor/

Issue is that scenario simulator uses a hardcopy of this repository with some modifications.

• https://github.com/tier4/scenario_simulator_v2/tree/master/external/zmqpp_vendor

Current https://github.com/tier4/zmqpp_vendor uses:
https://github.com/tier4/zmqpp_vendor/blob/225354f54fcca56fca6279a1d50c25ff373092e3/CMakeLists.txt#L16-L23

  include(ExternalProject)
  externalproject_add(cpp-zmqpp-ext
    GIT_REPOSITORY https://github.com/zeromq/zmqpp.git
    GIT_TAG master
    TIMEOUT 6000
    ${cmake_commands}
    CMAKE_ARGS ${cmake_configure_args}
  )

But the scenario sim uses it with:
https://github.com/tier4/scenario_simulator_v2/blob/463c03ced4e5c0ffa096697feda6177e8cd5dd67/external/zmqpp_vendor/CMakeLists.txt#L11-L16

ament_vendor(zmqpp_vendor
  SATISFIED ${zmqpp_FOUND}
  VCS_URL https://github.com/zeromq/zmqpp.git
  VCS_VERSION da73a138f290274cfd604b3f05a908956390a66e
  PATCHES patches/zmqpp_export.patch
)

And this change is done through:

• Use ament_vendor for zmqpp library tier4/scenario_simulator_v2#1634
  This destructive change (Install directory of zmqpp is changed).

So we should update the vendor package with the latest changes and use it from the ros build farm within the scenario simulator.

cc. @HansRobo @f0reachARR @hakuturu583

[HansRobo]

@xmfcx @mitsudome-r

The hard copy changes were originally planned to be rolled back into tier4/zmqpp_vendor, but the plan has no progress due to lack of strong need.
However, this is a good opportunity to resume them.

We have 2 changes in hardcopy.

1. Use of ament_vendor
   We configured the vendor package using tools from the ROS community.
   This is simply an implementation approach, and if any dependency issues arise, we can go back to ExternalProject.

2. Change to the install directory
   This change should be applied to tier4/zmqpp_vendor, but is not a migration blocker to the apt version.
   The original installation directory pollutes the include path, which may result in unintended source code being included in certain situations.
   Reference: Overlaying packages using CMake export targets can fail with merge install underlay ros2/ros2#1150

Overall, I should be able to remove the hard copy of zmqpp_vendor now and move to the apt version.
Would doing this solve your problem?

[xmfcx]

Thank you @HansRobo for your quick response!
We can keep using the ament_vendor() function from ament_cmake_vendor_package if we switch to the build farm.

So my recommended path would be like this:

1. Give @mitsudome-r and @HansRobo maintainer permissions for the https://github.com/tier4/zmqpp_vendor repository so we can maintain it quickly
2. Update the CI with ci: update humble and jazzy zmqpp_vendor#10
3. Apply the changes from scenario_simulator_v2 hardcopy into the tier4/zmqpp_vendor
4. Make a release in the build farm with the new version.
5. Remove the zmqpp_vendor hardcopy from the scenario_simulator_v2 and use zmqpp_vendor from the ros build farm debian repository.
6. Make a new release for the scenario_simulator_v2 and refer to it from the autoware main repository.

Benefits:

• This way, we don't need to build this dependency everytime we build scenario_simulator_v2
• Since zmqpp_vendor will have <buildtool_depend>ament_cmake_vendor_package</buildtool_depend>, it won't be a normal dependency for the apt, we can keep using this new somewhat cleaner method.

[xmfcx]

To be clear, when I copy paste the scenario sim version over the tier4 version, this is the diff:
(I omitted the changelog file)

diff --git a/.github/workflows/BloomRelease.yaml b/.github/workflows/BloomRelease.yaml
index ca09315..9791344 100644
--- a/.github/workflows/BloomRelease.yaml
+++ b/.github/workflows/BloomRelease.yaml
@@ -18,4 +18,4 @@ jobs:
           git_user: Masaya Kataoka
           git_email: masaya.kataoka@tier4.jp
           release_repository_push_url: https://github.com/${{ github.repository }}-release.git
-          open_pr: true
\ No newline at end of file
+          open_pr: true
diff --git a/.github/workflows/Release.yaml b/.github/workflows/Release.yaml
index bf2403c..2cdfd13 100644
--- a/.github/workflows/Release.yaml
+++ b/.github/workflows/Release.yaml
@@ -25,4 +25,4 @@ jobs:
           destination_branch: main
           pr_title: Release ${{ steps.create_release.outputs.version}}
           pr_body: close \#${{ github.event.issue.number }}
-          github_token: ${{ secrets.GITHUB_TOKEN }}
\ No newline at end of file
+          github_token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 75f3701..96ee0dc 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -6,36 +6,15 @@ if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
 endif()
 
 find_package(ament_cmake REQUIRED)
+find_package(ament_cmake_vendor_package REQUIRED)
 
-macro(build_zmqpp)
-  set(cmake_commands)
-  set(cmake_configure_args
-    -Wno-dev
-    -DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}_install)
+ament_vendor(zmqpp_vendor
+  SATISFIED ${zmqpp_FOUND}
+  VCS_URL https://github.com/zeromq/zmqpp.git
+  VCS_VERSION da73a138f290274cfd604b3f05a908956390a66e
+  PATCHES patches/zmqpp_export.patch
+)
 
-  include(ExternalProject)
-  externalproject_add(cpp-zmqpp-ext
-    GIT_REPOSITORY https://github.com/zeromq/zmqpp.git
-    GIT_TAG master
-    TIMEOUT 6000
-    ${cmake_commands}
-    CMAKE_ARGS ${cmake_configure_args}
-  )
-  install(
-    DIRECTORY
-      ${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}_install/
-    DESTINATION
-      ${CMAKE_INSTALL_PREFIX}
-  )
-endmacro()
+ament_export_dependencies(zmqpp)
 
-build_zmqpp()
-ament_export_include_directories(include)
-ament_export_libraries(zmqpp)
-
-if(BUILD_TESTING)
-  find_package(ament_lint_auto REQUIRED)
-  ament_lint_auto_find_test_dependencies()
-endif()
-
-ament_package()
+ament_package(CONFIG_EXTRAS zmqpp_vendor-extras.cmake)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 260a567..394c72c 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -11,4 +11,4 @@ be under the Apache 2 License, as dictated by that
    Notwithstanding the above, nothing herein shall supersede or modify
    the terms of any separate license agreement you may have executed
    with Licensor regarding such Contributions.
-~~~
\ No newline at end of file
+~~~
diff --git a/package.xml b/package.xml
index 2a55d17..3721ee1 100644
--- a/package.xml
+++ b/package.xml
@@ -2,13 +2,13 @@
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>zmqpp_vendor</name>
-  <version>0.0.2</version>
+  <version>18.2.1</version>
   <description>Vendor package for zmqpp</description>
   <maintainer email="masaya.kataoka@tier4.jp">Masaya Kataoka</maintainer>
   <license>Apache License 2.0</license>
-  <license>Mozilla Public License 2.0</license> <!-- sol2 is MPLv2 -->
 
   <buildtool_depend>ament_cmake</buildtool_depend>
+  <buildtool_depend>ament_cmake_vendor_package</buildtool_depend>
   <buildtool_depend>git</buildtool_depend>
 
   <depend>libzmq3-dev</depend>

[HansRobo]

@xmfcx
I agree with your recommended path.
An additional option would be to migrate tier4/zmqpp_vedor to autowarefoundation org, with the current maintainer's consent.
This would avoid permission management issues in this time and future

[xmfcx]

That would be the dream path 🤩

[xmfcx]

We would also update:

• https://github.com/ros/rosdistro/blob/db41777c636e3ef691984d9072fd59c845a4c9cb/humble/distribution.yaml#L13378-L13392
• move ros gbp ownership:
  • from: https://github.com/ros2-gbp/ros2-gbp-github-org/blob/e00bd197b4e83b8acc674342199275250402c8d4/tier4.tf#L15
  • to: https://github.com/ros2-gbp/ros2-gbp-github-org/blob/e00bd197b4e83b8acc674342199275250402c8d4/autoware.tf#L23

That's all.

[f0reachARR]

@xmfcx
I also agree with your proposed recommended path.

As @HansRobo mentioned, the hard copy in scenario_simulator_v2 has modifications regarding the install directory structure.
Here is the PR that attempted to rollback those changes in the zmqpp_vendor repository: autowarefoundation/zmqpp_vendor#9
The PR also describes the intent, and may be re-used.
However, since that PR uses ament_vendor, some modifications will be necessary to reuse it.

[xmfcx]

The repository has been moved by @mitsudome-r

• From: https://github.com/tier4/zmqpp_vendor
• To: https://github.com/autowarefoundation/zmqpp_vendor

Opened up the ROS Bloom related PRs:

• zmqpp_vendor: update the repository link ros/rosdistro#49111
• Move 'zmqpp_vendor' repo from 'tier4' to 'autoware' team ros2-gbp/ros2-gbp-github-org#950

[HansRobo]

We chose to avoid the error by stopping the use of ament_cmake_vendor_packages , but it seems that the root cause is being discussed in this issue.

ros-infrastructure/vcs2l#88