Skip to content

feat(core): Add support for building and running on manylinux_2_28; Add manylinux_2_28 dependency container images.#1041

Merged
kirkrodrigues merged 30 commits into
y-scope:mainfrom
jackluo923:manylinux_2_28
Jun 30, 2025
Merged

feat(core): Add support for building and running on manylinux_2_28; Add manylinux_2_28 dependency container images.#1041
kirkrodrigues merged 30 commits into
y-scope:mainfrom
jackluo923:manylinux_2_28

Conversation

@jackluo923
Copy link
Copy Markdown
Member

@jackluo923 jackluo923 commented Jun 26, 2025

Description

This PR adds support for building with manylinux_2_28. The manylinux project provides standardized and portable Linux build environments for Python binaries (wheels), ensuring wide compatibility across a variety of Linux distributions and platforms. By leveraging manylinux_2_28, we can produce CLP binaries for both x86_64 and aarch64 architectures that are easily distributable and require minimal external dependencies. These binaries are expected to be compatible with Debian 10+, Ubuntu 18.10+, Fedora 29+, CentOS/RHEL 8+, and many other Linux distributions.

Note1: The minimum required version of libcurl in CLP has been reduced from 7.68.0 to 7.61.1 to support building with manylinux_2_28. libcurl’s ABI is highly stable, so using an older version should not pose compatibility issues, and @LinZhihao-723 has cross-referenced API usage with docs and @jackluo923 has tested the functionality manually. If there is a need to support even lower versions in the future, we will revisit this decision.

Note2: LibArchive is now built with the -DENABLE_OPENSSL=OFF CMake option to prevent it from searching for static OpenSSL libraries during static linking, since they are no longer provided by most Linux distributions for security reasons.

This PR also restructures the containers docs since not all images are published now, and there is more information we want to add per container.

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a
    breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

  1. Generated both x86_64 and aarch64 binaries using the docker image.
# Build docker image
components/core/tools/docker-images/clp-env-base-manylinux_2_28_x86_64/build.sh
components/core/tools/docker-images/clp-env-base-manylinux_2_28_aarch64/build.sh

# Use docker image to build CLP on x86_64, aarch64 platform is similar
docker run -u $(id -u):$(id -g) -it -v /path/to/clp://clp clp-core-dependencies-manylinux_2_28_x86_64:dev /bin/bash
cd /clp
tools/scripts/deps-download/init.sh
task deps:core
cd components/core
mkdir build
cd build
cmake ../
make -j
  1. Validated that binary works on debian10 and ubuntu 20.04.
  2. Tested compression and decompression of local files and remote files:
# Compression Example
./clp-s c compressed https://raw.githubusercontent.com/openai/openai-cookbook/refs/heads/main/examples/data/toy_chat_fine_tuning.jsonl

# Decompression Example
./clp-s d compressed decompressed

Summary by CodeRabbit

  • New Features

    • Added new Dockerfiles and build scripts for manylinux_2_28 (ARM64 and x86_64) environments to streamline dependency management and image creation.
    • Introduced installation scripts for automating setup of prebuilt and source-built packages on manylinux_2_28 systems.
  • Documentation

    • Expanded and reorganized Docker container image documentation, providing detailed descriptions, usage instructions, and compatibility notes.
  • Bug Fixes

    • Improved static library linking detection for CentOS-based systems, ensuring AlmaLinux is correctly supported.
    • Relaxed minimum required version for CURL, increasing compatibility.
  • Chores

    • Updated build configuration to disable both Expat and OpenSSL for libarchive to prevent unwanted static linking.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants