diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 327b23d2206cc..67358353c3ec8 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1051,118 +1051,6 @@ updates: labels: - "dependabot" - "dependencies" - - directory: /qa/os/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/centos-6/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/centos-7/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/debian-8/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/debian-9/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/fedora-28/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/fedora-29/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/oel-6/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/oel-7/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/sles-12/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/ubuntu-1604/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/ubuntu-1804/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/windows-2012r2/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - - directory: /qa/os/windows-2016/ - open-pull-requests-limit: 1 - package-ecosystem: gradle - schedule: - interval: weekly - labels: - - "dependabot" - - "dependencies" - directory: /qa/remote-clusters/ open-pull-requests-limit: 1 package-ecosystem: gradle diff --git a/TESTING.md b/TESTING.md index 6d51e64dc9b1c..317ae0f0a174a 100644 --- a/TESTING.md +++ b/TESTING.md @@ -290,105 +290,6 @@ For example : You can launch a local OpenSearch cluster in debug mode following [Launching and debugging from an IDE](#launching-and-debugging-from-an-ide), and run your REST tests against that following [Running REST Tests Against An External Cluster](#running-rest-tests-against-an-external-cluster). -# Testing packaging - -The packaging tests use Vagrant virtual machines or cloud instances to verify that installing and running OpenSearch distributions works correctly on supported operating systems. These tests should really only be run on ephemeral systems because they’re destructive; that is, these tests install and remove packages and freely modify system settings, so you will probably regret it if you execute them on your development machine. - -When you run a packaging test, Gradle will set up the target VM and mount your repository directory in the VM. Once this is done, a Gradle task will issue a Vagrant command to run a **nested** Gradle task on the VM. This nested Gradle runs the actual "destructive" test classes. - -1. Install Virtual Box and Vagrant. - -2. (Optional) Install [vagrant-cachier](https://github.com/fgrehm/vagrant-cachier) to squeeze a bit more performance out of the process (Note: as of 2021, vagrant-cachier is unmaintained): - - vagrant plugin install vagrant-cachier - -3. You can run all the OS packaging tests with `./gradlew packagingTest`. This task includes our legacy `bats` tests. - - To run only the OS tests that are written in Java, run `.gradlew distroTest`, will cause Gradle to build the tar, zip, and deb packages and all the plugins. It will then run the tests on every available system. This will take a very long time. - - Fortunately, the various systems under test have their own Gradle tasks under `qa/os`. To find out what packaging combinations can be tested on a system, run the `tasks` task. For example: - - ./gradlew :qa:os:ubuntu-1804:tasks - - If you want a quick test of the tarball and RPM packaging for Centos 7, you would run: - - ./gradlew :qa:os:centos-7:distroTest.rpm :qa:os:centos-7:distroTest.linux-archive - -Note that if you interrupt Gradle in the middle of running these tasks, any boxes started will remain running, and you’ll have to stop them manually with `./gradlew --stop` or `vagrant halt`. - -All the regular vagrant commands should just work, so you can get a shell in a VM running trusty by running `vagrant up ubuntu-1604 --provider virtualbox && vagrant ssh ubuntu-1604`. - -These are the linux flavors supported, all of which we provide images for - -- ubuntu-1604 aka xenial -- ubuntu-1804 aka bionic beaver -- debian-8 aka jessie -- debian-9 aka stretch, the current debian stable distribution -- centos-6 -- centos-7 -- rhel-8 -- fedora-28 -- fedora-29 -- oel-6 aka Oracle Enterprise Linux 6 -- oel-7 aka Oracle Enterprise Linux 7 -- sles-12 -- opensuse-42 aka Leap - -We’re missing the following from the support matrix because there are no high quality boxes available in vagrant atlas: - -- sles-11 - -## Testing packaging on Windows - -The packaging tests also support Windows Server 2012R2 and Windows Server 2016. Unfortunately we’re not able to provide boxes for them in open source use because of licensing issues. Any Virtualbox image that has WinRM and Powershell enabled for remote users should work. - -Testing on Windows requires the [vagrant-winrm](https://github.com/criteo/vagrant-winrm) plugin. - - vagrant plugin install vagrant-winrm - -Specify the image IDs of the Windows boxes to gradle with the following project properties. They can be set in `~/.gradle/gradle.properties` such as - - vagrant.windows-2012r2.id=my-image-id - vagrant.windows-2016.id=another-image-id - -or passed on the command line such as `-Pvagrant.windows-2012r2.id=my-image-id` or `-Pvagrant.windows-2016=another-image-id` - -These properties are required for Windows support in all gradle tasks that handle packaging tests. Either or both may be specified. - -If you’re running vagrant commands outside of gradle, specify the Windows boxes with the environment variables. - -- `VAGRANT_WINDOWS_2012R2_BOX` -- `VAGRANT_WINDOWS_2016_BOX` - -## Testing VMs are disposable - -It’s important to think of VMs like cattle. If they become lame you just shoot them and let vagrant reprovision them. Say you’ve hosed your precise VM: - - vagrant ssh ubuntu-1604 -c 'sudo rm -rf /bin'; echo oops - -All you’ve got to do to get another one is - - vagrant destroy -f ubuntu-1604 && vagrant up ubuntu-1604 --provider virtualbox - -The whole process takes a minute and a half on a modern laptop, two and a half without vagrant-cachier. - -Some vagrant commands will work on all VMs at once: - - vagrant halt - vagrant destroy -f - -`vagrant up` would normally start all the VMs, but we’ve prevented that because that’d consume a ton of ram. - -## Iterating on packaging tests - -Because our packaging tests are capable of testing many combinations of OS (e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.) and so forth, it’s faster to develop against smaller subsets of the tests. For example, to run tests for the default archive distribution on Fedora 28: - - ./gradlew :qa:os:fedora-28:distroTest.linux-archive - -These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like non-OS tests can. For example: - - ./gradlew :qa:os:fedora-28:distroTest.linux-archive --tests "com.opensearch.packaging.test.ArchiveTests" - # Testing backwards compatibility Backwards compatibility tests exist to test upgrading from each supported version to the current version. diff --git a/Vagrantfile b/Vagrantfile deleted file mode 100644 index 28d5894664c2f..0000000000000 --- a/Vagrantfile +++ /dev/null @@ -1,499 +0,0 @@ -# -# SPDX-License-Identifier: Apache-2.0 -# -# The OpenSearch Contributors require contributions made to -# this file be licensed under the Apache-2.0 license or a -# compatible open source license. -# - -# -*- mode: ruby -*- -# vim: ft=ruby ts=2 sw=2 sts=2 et: - -# This Vagrantfile exists to test packaging. Read more about its use in the -# vagrant section in TESTING.md. - -# Licensed to Elasticsearch under one or more contributor -# license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright -# ownership. Elasticsearch licenses this file to you under -# the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# -# Modifications Copyright OpenSearch Contributors. See -# GitHub history for details. -# - -define_opts = { - autostart: false -}.freeze - -Vagrant.configure(2) do |config| - - config.vm.provider 'virtualbox' do |vbox| - # Give the box more memory and cpu because our tests are beasts! - vbox.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192) - vbox.cpus = Integer(ENV['VAGRANT_CPUS'] || 4) - - # see https://github.com/hashicorp/vagrant/issues/9524 - vbox.customize ["modifyvm", :id, "--audio", "none"] - end - - # Switch the default share for the project root from /vagrant to - # /opensearch because /vagrant is confusing when there is a project inside - # the opensearch project called vagrant.... - config.vm.synced_folder '.', '/vagrant', disabled: true - config.vm.synced_folder '.', '/opensearch' - # TODO: make these syncs work for windows!!! - config.vm.synced_folder "#{Dir.home}/.vagrant/gradle/caches/jars-3", "/root/.gradle/caches/jars-3", - create: true, - owner: "vagrant" - config.vm.synced_folder "#{Dir.home}/.vagrant/gradle/caches/modules-2", "/root/.gradle/caches/modules-2", - create: true, - owner: "vagrant" - config.vm.synced_folder "#{Dir.home}/.gradle/wrapper", "/root/.gradle/wrapper", - create: true, - owner: "vagrant" - - # Expose project directory. Note that VAGRANT_CWD may not be the same as Dir.pwd - PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd - config.vm.synced_folder PROJECT_DIR, '/project' - - 'ubuntu-1604'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/ubuntu-16.04-x86_64' - deb_common config, box, extra: <<-SHELL - # Install Jayatana so we can work around it being present. - [ -f /usr/share/java/jayatanaag.jar ] || install jayatana - SHELL - ubuntu_docker config - end - end - 'ubuntu-1804'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/ubuntu-18.04-x86_64' - deb_common config, box, extra: <<-SHELL - # Install Jayatana so we can work around it being present. - [ -f /usr/share/java/jayatanaag.jar ] || install jayatana - SHELL - ubuntu_docker config - end - end - # Wheezy's backports don't contain Openjdk 8 and the backflips - # required to get the sun jdk on there just aren't worth it. We have - # jessie and stretch for testing debian and it works fine. - 'debian-8'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/debian-8-x86_64' - deb_common config, box, extra: <<-SHELL - # this sometimes gets a bad ip, and doesn't appear to be needed - rm -f /etc/apt/sources.list.d/http_debian_net_debian.list - SHELL - end - end - 'debian-9'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/debian-9-x86_64' - deb_common config, box - deb_docker config - end - end - 'centos-6'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/centos-6-x86_64' - rpm_common config, box - end - end - 'centos-7'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/centos-7-x86_64' - rpm_common config, box - rpm_docker config - end - end - 'oel-6'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/oraclelinux-6-x86_64' - rpm_common config, box - end - end - 'oel-7'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/oraclelinux-7-x86_64' - rpm_common config, box - end - end - 'fedora-28'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/fedora-28-x86_64' - dnf_common config, box - dnf_docker config - end - end - 'fedora-29'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/fedora-29-x86_64' - dnf_common config, box - dnf_docker config - end - end - 'sles-12'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/sles-12-x86_64' - sles_common config, box - end - end - 'rhel-8'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = 'elastic/rhel-8-x86_64' - rpm_common config, box - end - end - - windows_2012r2_box = ENV['VAGRANT_WINDOWS_2012R2_BOX'] - if windows_2012r2_box && windows_2012r2_box.empty? == false - 'windows-2012r2'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = windows_2012r2_box - windows_common config, box - end - end - end - - windows_2016_box = ENV['VAGRANT_WINDOWS_2016_BOX'] - if windows_2016_box && windows_2016_box.empty? == false - 'windows-2016'.tap do |box| - config.vm.define box, define_opts do |config| - config.vm.box = windows_2016_box - windows_common config, box - end - end - end -end - -def deb_common(config, name, extra: '') - # http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html - config.vm.provision 'fix-no-tty', type: 'shell' do |s| - s.privileged = false - s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile" - end - extra_with_lintian = <<-SHELL - #{extra} - install lintian - SHELL - linux_common( - config, - name, - update_command: 'apt-get update', - update_tracking_file: '/var/cache/apt/archives/last_update', - install_command: 'apt-get install -y --force-yes', - extra: extra_with_lintian - ) -end - -def ubuntu_docker(config) - config.vm.provision 'install Docker using apt', type: 'shell', inline: <<-SHELL - # Install packages to allow apt to use a repository over HTTPS - apt-get install -y --force-yes \ - apt-transport-https \ - ca-certificates \ - curl \ - gnupg2 \ - software-properties-common - - # Add Docker’s official GPG key - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - - - # Set up the stable Docker repository - add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) \ - stable" - - # Install Docker. Unlike Fedora and CentOS, this also start the daemon. - apt-get update - apt-get install -y --force-yes docker-ce docker-ce-cli containerd.io - - # Add vagrant to the Docker group, so that it can run commands - usermod -aG docker vagrant - - # Enable IPv4 forwarding - sed -i '/net.ipv4.ip_forward/s/^#//' /etc/sysctl.conf - systemctl restart networking - SHELL -end - - -def deb_docker(config) - config.vm.provision 'install Docker using apt', type: 'shell', inline: <<-SHELL - # Install packages to allow apt to use a repository over HTTPS - apt-get install -y --force-yes \ - apt-transport-https \ - ca-certificates \ - curl \ - gnupg2 \ - software-properties-common - - # Add Docker’s official GPG key - curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - - - # Set up the stable Docker repository - add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/debian \ - $(lsb_release -cs) \ - stable" - - # Install Docker. Unlike Fedora and CentOS, this also start the daemon. - apt-get update - apt-get install -y --force-yes docker-ce docker-ce-cli containerd.io - - # Add vagrant to the Docker group, so that it can run commands - usermod -aG docker vagrant - SHELL -end - -def rpm_common(config, name) - linux_common( - config, - name, - update_command: 'yum check-update', - update_tracking_file: '/var/cache/yum/last_update', - install_command: 'yum install -y' - ) -end - -def rpm_docker(config) - config.vm.provision 'install Docker using yum', type: 'shell', inline: <<-SHELL - # Install prerequisites - yum install -y yum-utils device-mapper-persistent-data lvm2 - - # Add repository - yum-config-manager -y --add-repo https://download.docker.com/linux/centos/docker-ce.repo - - # Install Docker - yum install -y docker-ce docker-ce-cli containerd.io - - # Start Docker - systemctl enable --now docker - - # Add vagrant to the Docker group, so that it can run commands - usermod -aG docker vagrant - SHELL -end - -def dnf_common(config, name) - # Autodetect doesn't work.... - if Vagrant.has_plugin?('vagrant-cachier') - config.cache.auto_detect = false - config.cache.enable :generic, { :cache_dir => '/var/cache/dnf' } - end - linux_common( - config, - name, - update_command: 'dnf check-update', - update_tracking_file: '/var/cache/dnf/last_update', - install_command: 'dnf install -y', - install_command_retries: 5 - ) -end - -def dnf_docker(config) - config.vm.provision 'install Docker using dnf', type: 'shell', inline: <<-SHELL - # Install prerequisites - dnf -y install dnf-plugins-core - - # Add repository - dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo - - # Install Docker - dnf install -y docker-ce docker-ce-cli containerd.io - - # Start Docker - systemctl enable --now docker - - # Add vagrant to the Docker group, so that it can run commands - usermod -aG docker vagrant - SHELL -end - -def suse_common(config, name, extra: '') - linux_common( - config, - name, - update_command: 'zypper --non-interactive list-updates', - update_tracking_file: '/var/cache/zypp/packages/last_update', - install_command: 'zypper --non-interactive --quiet install --no-recommends', - extra: extra - ) -end - -def sles_common(config, name) - extra = <<-SHELL - zypper rr systemsmanagement_puppet puppetlabs-pc1 devel_tools_scm - zypper ar http://download.opensuse.org/distribution/12.3/repo/oss/ oss - zypper --non-interactive --gpg-auto-import-keys refresh - zypper --non-interactive install git-core - # choose to "ignore some dependencies" of expect, which has a problem with tcl... - zypper --non-interactive install --force-resolution expect - SHELL - suse_common config, name, extra: extra -end - -# Configuration needed for all linux boxes -# @param config Vagrant's config object. Required. -# @param name [String] The box name. Required. -# @param update_command [String] The command used to update the package -# manager. Required. Think `apt-get update`. -# @param update_tracking_file [String] The location of the file tracking the -# last time the update command was run. Required. Should be in a place that -# is cached by vagrant-cachier. -# @param install_command [String] The command used to install a package. -# Required. Think `apt-get install #{package}`. -# @param install_command_retries [Integer] Number of times to retry -# a failed install command -# @param extra [String] Additional script to run before installing -# dependencies -# -def linux_common(config, - name, - update_command: 'required', - update_tracking_file: 'required', - install_command: 'required', - install_command_retries: 0, - extra: '') - - raise ArgumentError, 'update_command is required' if update_command == 'required' - raise ArgumentError, 'update_tracking_file is required' if update_tracking_file == 'required' - raise ArgumentError, 'install_command is required' if install_command == 'required' - - if Vagrant.has_plugin?('vagrant-cachier') - config.cache.scope = :box - end - - config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL - touch /etc/is_vagrant_vm - touch /is_vagrant_vm # for consistency between linux and windows - SHELL - - # This prevents leftovers from previous tests using the - # same VM from messing up the current test - config.vm.provision 'clean opensearch installs in tmp', run: 'always', type: 'shell', inline: <<-SHELL - rm -rf /tmp/opensearch* - SHELL - - sh_set_prompt config, name - sh_install_deps( - config, - update_command, - update_tracking_file, - install_command, - install_command_retries, - extra - ) -end - -# Sets up a consistent prompt for all users. Or tries to. The VM might -# contain overrides for root and vagrant but this attempts to work around -# them by re-source-ing the standard prompt file. -def sh_set_prompt(config, name) - config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL - cat \<\ /etc/profile.d/opensearch_prompt.sh -export PS1='#{name}:\\w$ ' -PROMPT - grep 'source /etc/profile.d/opensearch_prompt.sh' ~/.bashrc | - cat \<\> ~/.bashrc -# Replace the standard prompt with a consistent one -source /etc/profile.d/opensearch_prompt.sh -SOURCE_PROMPT - grep 'source /etc/profile.d/opensearch_prompt.sh' ~vagrant/.bashrc | - cat \<\> ~vagrant/.bashrc -# Replace the standard prompt with a consistent one -source /etc/profile.d/opensearch_prompt.sh -SOURCE_PROMPT - SHELL -end - -def sh_install_deps(config, - update_command, - update_tracking_file, - install_command, - install_command_retries, - extra) - config.vm.provision 'install dependencies', type: 'shell', inline: <<-SHELL - set -e - set -o pipefail - - # Retry install command up to $2 times, if failed - retry_installcommand() { - n=0 - while true; do - #{install_command} $1 && break - let n=n+1 - if [ $n -ge $2 ]; then - echo "==> Exhausted retries to install $1" - return 1 - fi - echo "==> Retrying installing $1, attempt $((n+1))" - # Add a small delay to increase chance of metalink providing updated list of mirrors - sleep 5 - done - } - - installed() { - command -v $1 2>&1 >/dev/null - } - - install() { - # Only apt-get update if we haven't in the last day - if [ ! -f #{update_tracking_file} ] || [ "x$(find #{update_tracking_file} -mtime +0)" == "x#{update_tracking_file}" ]; then - echo "==> Updating repository" - #{update_command} || true - touch #{update_tracking_file} - fi - echo "==> Installing $1" - if [ #{install_command_retries} -eq 0 ] - then - #{install_command} $1 - else - retry_installcommand $1 #{install_command_retries} - fi - } - - ensure() { - installed $1 || install $1 - } - - #{extra} - - ensure tar - ensure curl - ensure unzip - ensure rsync - ensure expect - - cat \<\ /etc/sudoers.d/opensearch_vars -Defaults env_keep += "JAVA_HOME" -Defaults env_keep += "SYSTEM_JAVA_HOME" -Defaults env_keep += "OPENSEARCH_JAVA_HOME" -SUDOERS_VARS - chmod 0440 /etc/sudoers.d/opensearch_vars - SHELL -end - -def windows_common(config, name) - config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL - $ErrorActionPreference = "Stop" - $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }' - $ps_prompt | Out-File $PsHome/Microsoft.PowerShell_profile.ps1 - SHELL -end diff --git a/buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java b/buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java deleted file mode 100644 index 66aa0ac3ec4ea..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java +++ /dev/null @@ -1,496 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.test; - -import org.opensearch.gradle.Architecture; -import org.opensearch.gradle.DistributionDownloadPlugin; -import org.opensearch.gradle.JavaPackageType; -import org.opensearch.gradle.Jdk; -import org.opensearch.gradle.JdkDownloadPlugin; -import org.opensearch.gradle.OpenSearchDistribution; -import org.opensearch.gradle.SystemPropertyCommandLineArgumentProvider; -import org.opensearch.gradle.Version; -import org.opensearch.gradle.VersionProperties; -import org.opensearch.gradle.docker.DockerSupportPlugin; -import org.opensearch.gradle.docker.DockerSupportService; -import org.opensearch.gradle.info.BuildParams; -import org.opensearch.gradle.internal.InternalDistributionDownloadPlugin; -import org.opensearch.gradle.util.GradleUtils; -import org.opensearch.gradle.util.Util; -import org.opensearch.gradle.vagrant.VagrantBasePlugin; -import org.opensearch.gradle.vagrant.VagrantExtension; -import org.opensearch.gradle.vagrant.VagrantMachine; -import org.gradle.api.Action; -import org.gradle.api.NamedDomainObjectContainer; -import org.gradle.api.Plugin; -import org.gradle.api.Project; -import org.gradle.api.Task; -import org.gradle.api.artifacts.Configuration; -import org.gradle.api.artifacts.dsl.DependencyHandler; -import org.gradle.api.plugins.JavaBasePlugin; -import org.gradle.api.provider.Provider; -import org.gradle.api.specs.Specs; -import org.gradle.api.tasks.Copy; -import org.gradle.api.tasks.TaskProvider; -import org.gradle.api.tasks.testing.Test; - -import java.io.File; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.HashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Map.Entry; -import java.util.Set; -import java.util.function.Supplier; -import java.util.stream.Stream; - -public class DistroTestPlugin implements Plugin { - private static final String SYSTEM_JDK_VERSION = "25.0.4.1+1"; - private static final String SYSTEM_JDK_VENDOR = "adoptium"; - private static final String GRADLE_JDK_VERSION = "25.0.4.1+1"; - private static final String GRADLE_JDK_VENDOR = "adoptium"; - - // all distributions used by distro tests. this is temporary until tests are per distribution - private static final String EXAMPLE_PLUGIN_CONFIGURATION = "examplePlugin"; - private static final String IN_VM_SYSPROP = "tests.inVM"; - private static final String DISTRIBUTION_SYSPROP = "tests.distribution"; - private static final String BWC_DISTRIBUTION_SYSPROP = "tests.bwc-distribution"; - private static final String EXAMPLE_PLUGIN_SYSPROP = "tests.example-plugin"; - - @Override - public void apply(Project project) { - project.getRootProject().getPluginManager().apply(DockerSupportPlugin.class); - project.getPlugins().apply(InternalDistributionDownloadPlugin.class); - project.getPlugins().apply(JdkDownloadPlugin.class); - project.getPluginManager().apply("opensearch.build"); - - Provider dockerSupport = GradleUtils.getBuildService( - project.getGradle().getSharedServices(), - DockerSupportPlugin.DOCKER_SUPPORT_SERVICE_NAME - ); - - // TODO: it would be useful to also have the SYSTEM_JAVA_HOME setup in the root project, so that running from GCP only needs - // a java for gradle to run, and the tests are self sufficient and consistent with the java they use - NamedDomainObjectContainer allDistributions = DistributionDownloadPlugin.getContainer(project); - List testDistributions = configureDistributions(project); - - Map> lifecycleTasks = lifecycleTasks(project, "destructiveDistroTest"); - Map> versionTasks = versionTasks(project, "destructiveDistroUpgradeTest"); - TaskProvider destructiveDistroTest = project.getTasks().register("destructiveDistroTest"); - - Configuration examplePlugin = configureExamplePlugin(project); - - List> windowsTestTasks = new ArrayList<>(); - Map>> linuxTestTasks = new HashMap<>(); - Map>> upgradeTestTasks = new HashMap<>(); - Map> depsTasks = new HashMap<>(); - for (OpenSearchDistribution distribution : testDistributions) { - String taskname = destructiveDistroTestTaskName(distribution); - TaskProvider depsTask = project.getTasks().register(taskname + "#deps"); - depsTask.configure(t -> t.dependsOn(distribution, examplePlugin)); - depsTasks.put(taskname, depsTask); - // Avoid duplicate tasks such as docker registered in lifecycleTasks - if (project.getTasksByName(taskname, false).isEmpty()) { - TaskProvider destructiveTask = configureTestTask(project, taskname, distribution, t -> { - t.onlyIf(t2 -> distribution.isDocker() == false || dockerSupport.get().getDockerAvailability().isAvailable); - addSysprop(t, DISTRIBUTION_SYSPROP, distribution::getFilepath); - addSysprop(t, EXAMPLE_PLUGIN_SYSPROP, () -> examplePlugin.getSingleFile().toString()); - t.exclude("**/PackageUpgradeTests.class"); - }, depsTask); - if (distribution.getPlatform() == OpenSearchDistribution.Platform.WINDOWS) { - windowsTestTasks.add(destructiveTask); - } else { - linuxTestTasks.computeIfAbsent(distribution.getType(), k -> new ArrayList<>()).add(destructiveTask); - } - destructiveDistroTest.configure(t -> t.dependsOn(destructiveTask)); - lifecycleTasks.get(distribution.getType()).configure(t -> t.dependsOn(destructiveTask)); - } - - if ((distribution.getType() == OpenSearchDistribution.Type.DEB || distribution.getType() == OpenSearchDistribution.Type.RPM) - && distribution.getBundledJdk() != JavaPackageType.NONE) { - for (Version version : BuildParams.getBwcVersions().getIndexCompatible()) { - if (version.before("6.3.0")) { - continue; // before opening xpack - } - final OpenSearchDistribution bwcDistro; - if (version.equals(Version.fromString(distribution.getVersion()))) { - // this is the same as the distribution we are testing - bwcDistro = distribution; - } else { - bwcDistro = createDistro( - allDistributions, - distribution.getArchitecture(), - distribution.getType(), - distribution.getPlatform(), - distribution.getBundledJdk(), - version.toString() - ); - - } - String upgradeTaskname = destructiveDistroUpgradeTestTaskName(distribution, version.toString()); - TaskProvider upgradeDepsTask = project.getTasks().register(upgradeTaskname + "#deps"); - upgradeDepsTask.configure(t -> t.dependsOn(distribution, bwcDistro)); - depsTasks.put(upgradeTaskname, upgradeDepsTask); - TaskProvider upgradeTest = configureTestTask(project, upgradeTaskname, distribution, t -> { - addSysprop(t, DISTRIBUTION_SYSPROP, distribution::getFilepath); - addSysprop(t, BWC_DISTRIBUTION_SYSPROP, bwcDistro::getFilepath); - t.include("**/PackageUpgradeTests.class"); - }, upgradeDepsTask); - versionTasks.get(version.toString()).configure(t -> t.dependsOn(upgradeTest)); - upgradeTestTasks.computeIfAbsent(version.toString(), k -> new ArrayList<>()).add(upgradeTest); - } - } - } - - // setup jdks used by no-jdk tests, and by gradle executing - TaskProvider linuxGradleJdk = createJdk(project, "gradle", GRADLE_JDK_VENDOR, GRADLE_JDK_VERSION, "linux", "x64"); - TaskProvider linuxSystemJdk = createJdk(project, "system", SYSTEM_JDK_VENDOR, SYSTEM_JDK_VERSION, "linux", "x64"); - TaskProvider windowsGradleJdk = createJdk(project, "gradle", GRADLE_JDK_VENDOR, GRADLE_JDK_VERSION, "windows", "x64"); - TaskProvider windowsSystemJdk = createJdk(project, "system", SYSTEM_JDK_VENDOR, SYSTEM_JDK_VERSION, "windows", "x64"); - - project.subprojects(vmProject -> { - vmProject.getPluginManager().apply(VagrantBasePlugin.class); - TaskProvider gradleJdk = isWindows(vmProject) ? windowsGradleJdk : linuxGradleJdk; - TaskProvider systemJdk = isWindows(vmProject) ? windowsSystemJdk : linuxSystemJdk; - configureVM(vmProject, gradleJdk, systemJdk); - List vmDependencies = Arrays.asList( - gradleJdk, - systemJdk, - project.getConfigurations().getByName("testRuntimeClasspath") - ); - - Map> vmLifecyleTasks = lifecycleTasks(vmProject, "distroTest"); - Map> vmVersionTasks = versionTasks(vmProject, "distroUpgradeTest"); - TaskProvider distroTest = vmProject.getTasks().register("distroTest"); - - // windows boxes get windows distributions, and linux boxes get linux distributions - if (isWindows(vmProject)) { - configureVMWrapperTasks(vmProject, windowsTestTasks, depsTasks, wrapperTask -> { - vmLifecyleTasks.get(OpenSearchDistribution.Type.ARCHIVE).configure(t -> t.dependsOn(wrapperTask)); - }, vmDependencies); - } else { - for (Entry>> entry : linuxTestTasks.entrySet()) { - OpenSearchDistribution.Type type = entry.getKey(); - TaskProvider vmLifecycleTask = vmLifecyleTasks.get(type); - configureVMWrapperTasks(vmProject, entry.getValue(), depsTasks, wrapperTask -> { - vmLifecycleTask.configure(t -> t.dependsOn(wrapperTask)); - - // Only VM sub-projects that are specifically opted-in to testing Docker should - // have the Docker task added as a dependency. Although we control whether Docker - // is installed in the VM via `Vagrantfile` and we could auto-detect its presence - // in the VM, the test tasks e.g. `destructiveDistroTest.default-docker` are defined - // on the host during Gradle's configuration phase and not in the VM, so - // auto-detection doesn't work. - // - // The shouldTestDocker property could be null, hence we use Boolean.TRUE.equals() - boolean shouldExecute = (type != OpenSearchDistribution.Type.DOCKER) - || Boolean.TRUE.equals(vmProject.findProperty("shouldTestDocker")); - - if (shouldExecute) { - distroTest.configure(t -> t.dependsOn(wrapperTask)); - } - }, vmDependencies); - } - - for (Entry>> entry : upgradeTestTasks.entrySet()) { - String version = entry.getKey(); - TaskProvider vmVersionTask = vmVersionTasks.get(version); - configureVMWrapperTasks( - vmProject, - entry.getValue(), - depsTasks, - wrapperTask -> { vmVersionTask.configure(t -> t.dependsOn(wrapperTask)); }, - vmDependencies - ); - } - } - }); - } - - private static Map> lifecycleTasks(Project project, String taskPrefix) { - Map> lifecyleTasks = new HashMap<>(); - - lifecyleTasks.put(OpenSearchDistribution.Type.DOCKER, project.getTasks().register(taskPrefix + ".docker")); - lifecyleTasks.put(OpenSearchDistribution.Type.ARCHIVE, project.getTasks().register(taskPrefix + ".archives")); - lifecyleTasks.put(OpenSearchDistribution.Type.DEB, project.getTasks().register(taskPrefix + ".packages")); - lifecyleTasks.put(OpenSearchDistribution.Type.RPM, lifecyleTasks.get(OpenSearchDistribution.Type.DEB)); - - return lifecyleTasks; - } - - private static Map> versionTasks(Project project, String taskPrefix) { - Map> versionTasks = new HashMap<>(); - - for (Version version : BuildParams.getBwcVersions().getIndexCompatible()) { - versionTasks.put(version.toString(), project.getTasks().register(taskPrefix + ".v" + version)); - } - - return versionTasks; - } - - private static TaskProvider createJdk( - Project project, - String purpose, - String vendor, - String version, - String platform, - String architecture - ) { - Jdk jdk = JdkDownloadPlugin.getContainer(project).create(platform + "-" + purpose); - jdk.setVendor(vendor); - jdk.setVersion(version); - jdk.setPlatform(platform); - jdk.setArchitecture(architecture); - - String taskname = "copy" + Util.capitalize(platform) + Util.capitalize(purpose) + "Jdk"; - TaskProvider copyTask = project.getTasks().register(taskname, Copy.class); - copyTask.configure(t -> { - t.from(jdk); - t.into(new File(project.getBuildDir(), "jdks/" + platform + "-" + architecture + "-" + vendor + "-" + version)); - }); - return copyTask; - } - - private static void configureVM(Project project, TaskProvider gradleJdkProvider, TaskProvider systemJdkProvider) { - String box = project.getName(); - - // setup VM used by these tests - VagrantExtension vagrant = project.getExtensions().getByType(VagrantExtension.class); - vagrant.setBox(box); - vagrant.vmEnv("SYSTEM_JAVA_HOME", convertPath(project, vagrant, systemJdkProvider, "", "")); - // set java home for gradle to use. package tests will overwrite/remove this for each test case - vagrant.vmEnv("JAVA_HOME", convertPath(project, vagrant, gradleJdkProvider, "", "")); - if (System.getenv("JENKINS_URL") != null) { - Stream.of("JOB_NAME", "JENKINS_URL", "BUILD_NUMBER", "BUILD_URL").forEach(name -> vagrant.vmEnv(name, System.getenv(name))); - } - vagrant.setIsWindowsVM(isWindows(project)); - } - - private static Object convertPath( - Project project, - VagrantExtension vagrant, - TaskProvider jdkProvider, - String additionaLinux, - String additionalWindows - ) { - return Util.toStringable(() -> { - String hostPath = jdkProvider.get().getDestinationDir().toString(); - if (vagrant.isWindowsVM()) { - return VagrantMachine.convertWindowsPath(project, hostPath) + additionalWindows; - } else { - return VagrantMachine.convertLinuxPath(project, hostPath) + additionaLinux; - } - }); - } - - private static Configuration configureExamplePlugin(Project project) { - Configuration examplePlugin = project.getConfigurations().create(EXAMPLE_PLUGIN_CONFIGURATION); - DependencyHandler deps = project.getDependencies(); - Map examplePluginProject = new HashMap() { - { - put("path", ":example-plugins:custom-settings"); - put("configuration", "zip"); - } - }; - deps.add(EXAMPLE_PLUGIN_CONFIGURATION, deps.project(examplePluginProject)); - return examplePlugin; - } - - private static void configureVMWrapperTasks( - Project project, - List> destructiveTasks, - Map> depsTasks, - Action> configure, - Object... additionalDeps - ) { - for (TaskProvider destructiveTask : destructiveTasks) { - String destructiveTaskName = destructiveTask.getName(); - String taskname = destructiveTaskName.substring("destructive".length()); - taskname = taskname.substring(0, 1).toLowerCase(Locale.ROOT) + taskname.substring(1); - TaskProvider vmTask = project.getTasks().register(taskname, GradleDistroTestTask.class, t -> { - t.setGroup(JavaBasePlugin.VERIFICATION_GROUP); - t.setDescription("Runs " + destructiveTaskName.split("\\.", 2)[1] + " tests within vagrant"); - t.setTaskName(destructiveTaskName); - t.extraArg("-D'" + IN_VM_SYSPROP + "'"); - t.dependsOn(depsTasks.get(destructiveTaskName)); - t.dependsOn(additionalDeps); - }); - configure.execute(vmTask); - } - } - - private static TaskProvider configureTestTask( - Project project, - String taskname, - OpenSearchDistribution distribution, - Action configure, - Object... deps - ) { - return project.getTasks().register(taskname, Test.class, t -> { - // Only run tests for the current architecture - t.onlyIf(t3 -> distribution.getArchitecture() == Architecture.current()); - t.getOutputs().doNotCacheIf("Build cache is disabled for packaging tests", Specs.satisfyAll()); - t.setMaxParallelForks(1); - t.setWorkingDir(project.getProjectDir()); - if (System.getProperty(IN_VM_SYSPROP) == null) { - t.dependsOn(deps); - } - configure.execute(t); - }); - } - - private List configureDistributions(Project project) { - NamedDomainObjectContainer distributions = DistributionDownloadPlugin.getContainer(project); - List currentDistros = new ArrayList<>(); - - for (Architecture architecture : Architecture.values()) { - for (OpenSearchDistribution.Type type : Arrays.asList( - OpenSearchDistribution.Type.DEB, - OpenSearchDistribution.Type.RPM, - OpenSearchDistribution.Type.DOCKER - )) { - for (JavaPackageType bundledJdk : Set.of(JavaPackageType.NONE, JavaPackageType.JDK)) { - if (bundledJdk == JavaPackageType.NONE) { - // We'll never publish an ARM (arm64) build without a bundled JDK. - if (architecture == Architecture.ARM64) { - continue; - } - // All our Docker images include a bundled JDK so it doesn't make sense to test without one. - if (type == OpenSearchDistribution.Type.DOCKER) { - continue; - } - } - - currentDistros.add( - createDistro(distributions, architecture, type, null, bundledJdk, VersionProperties.getOpenSearch()) - ); - } - } - } - - for (Architecture architecture : Architecture.values()) { - for (OpenSearchDistribution.Platform platform : Arrays.asList( - OpenSearchDistribution.Platform.LINUX, - OpenSearchDistribution.Platform.WINDOWS - )) { - for (JavaPackageType bundledJdk : Set.of(JavaPackageType.NONE, JavaPackageType.JDK)) { - if (bundledJdk == JavaPackageType.NONE && architecture != Architecture.X64) { - // We will never publish distributions for non-x86 (amd64) platforms - // without a bundled JDK - continue; - } - - currentDistros.add( - createDistro( - distributions, - architecture, - OpenSearchDistribution.Type.ARCHIVE, - platform, - bundledJdk, - VersionProperties.getOpenSearch() - ) - ); - } - } - } - - return currentDistros; - } - - private static OpenSearchDistribution createDistro( - NamedDomainObjectContainer distributions, - Architecture architecture, - OpenSearchDistribution.Type type, - OpenSearchDistribution.Platform platform, - JavaPackageType bundledJdk, - String version - ) { - String name = distroId(type, platform, bundledJdk, architecture) + "-" + version; - boolean isDocker = type == OpenSearchDistribution.Type.DOCKER; - OpenSearchDistribution distro = distributions.create(name, d -> { - d.setArchitecture(architecture); - d.setType(type); - if (type == OpenSearchDistribution.Type.ARCHIVE) { - d.setPlatform(platform); - } - if (isDocker == false) { - d.setBundledJdk(bundledJdk); - } - d.setVersion(version); - }); - - // Allow us to gracefully omit building Docker distributions if Docker is not available on the system. - // In such a case as we can't build the Docker images we'll simply skip the corresponding tests. - if (isDocker) { - distro.setFailIfUnavailable(false); - } - - return distro; - } - - // return true if the project is for a windows VM, false otherwise - private static boolean isWindows(Project project) { - return project.getName().contains("windows"); - } - - private static String distroId( - OpenSearchDistribution.Type type, - OpenSearchDistribution.Platform platform, - JavaPackageType bundledJdk, - Architecture architecture - ) { - return (type == OpenSearchDistribution.Type.ARCHIVE ? platform + "-" : "") + type + (bundledJdk != JavaPackageType.NONE - ? (bundledJdk == JavaPackageType.JDK ? "" : "-jre") - : "-no-jdk") + (architecture == Architecture.X64 ? "" : "-" + architecture.toString().toLowerCase()); - } - - private static String destructiveDistroTestTaskName(OpenSearchDistribution distro) { - OpenSearchDistribution.Type type = distro.getType(); - return "destructiveDistroTest." + distroId(type, distro.getPlatform(), distro.getBundledJdk(), distro.getArchitecture()); - } - - private static String destructiveDistroUpgradeTestTaskName(OpenSearchDistribution distro, String bwcVersion) { - OpenSearchDistribution.Type type = distro.getType(); - return "destructiveDistroUpgradeTest.v" - + bwcVersion - + "." - + distroId(type, distro.getPlatform(), distro.getBundledJdk(), distro.getArchitecture()); - } - - private static void addSysprop(Test task, String sysprop, Supplier valueSupplier) { - SystemPropertyCommandLineArgumentProvider props = task.getExtensions().getByType(SystemPropertyCommandLineArgumentProvider.class); - props.systemProperty(sysprop, valueSupplier); - } -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/test/GradleDistroTestTask.java b/buildSrc/src/main/java/org/opensearch/gradle/test/GradleDistroTestTask.java deleted file mode 100644 index caac3ede98588..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/test/GradleDistroTestTask.java +++ /dev/null @@ -1,113 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.test; - -import org.opensearch.gradle.vagrant.VagrantMachine; -import org.opensearch.gradle.vagrant.VagrantShellTask; -import org.gradle.api.Project; -import org.gradle.api.tasks.Input; -import org.gradle.api.tasks.options.Option; - -import javax.inject.Inject; - -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; - -/** - * Run a gradle task of the current build, within the configured vagrant VM. - */ -public class GradleDistroTestTask extends VagrantShellTask { - - private String taskName; - private String testClass; - private List extraArgs = new ArrayList<>(); - private final Project project; - - @Inject - public GradleDistroTestTask(Project project) { - super(project); - this.project = project; - } - - public void setTaskName(String taskName) { - this.taskName = taskName; - } - - @Input - public String getTaskName() { - return taskName; - } - - @Option(option = "tests", description = "Sets test class or method name to be included, '*' is supported.") - public void setTestClass(String testClass) { - this.testClass = testClass; - } - - @Input - public List getExtraArgs() { - return extraArgs; - } - - public void extraArg(String arg) { - this.extraArgs.add(arg); - } - - @Override - protected List getWindowsScript() { - return getScript(true); - } - - @Override - protected List getLinuxScript() { - return getScript(false); - } - - private List getScript(boolean isWindows) { - String cacheDir = project.getBuildDir() + "/gradle-cache"; - StringBuilder line = new StringBuilder(); - line.append(isWindows ? "& .\\gradlew " : "./gradlew "); - line.append(taskName); - line.append(" --project-cache-dir "); - line.append(isWindows ? VagrantMachine.convertWindowsPath(project, cacheDir) : VagrantMachine.convertLinuxPath(project, cacheDir)); - line.append(" -S"); - line.append(" --parallel"); - line.append(" -D'org.gradle.logging.level'=" + project.getGradle().getStartParameter().getLogLevel()); - if (testClass != null) { - line.append(" --tests="); - line.append(testClass); - } - extraArgs.stream().map(s -> " " + s).forEach(line::append); - return Collections.singletonList(line.toString()); - } -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantBasePlugin.java b/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantBasePlugin.java deleted file mode 100644 index db1d5d5c89bbb..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantBasePlugin.java +++ /dev/null @@ -1,180 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.vagrant; - -import org.opensearch.gradle.ReaperPlugin; -import org.opensearch.gradle.ReaperService; -import org.gradle.api.Plugin; -import org.gradle.api.Project; -import org.gradle.api.Task; -import org.gradle.api.execution.TaskActionListener; -import org.gradle.api.execution.TaskExecutionListener; -import org.gradle.api.tasks.TaskState; -import org.gradle.process.ExecOperations; - -import javax.inject.Inject; - -import java.io.ByteArrayOutputStream; -import java.nio.charset.StandardCharsets; -import java.util.List; -import java.util.function.Consumer; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -public class VagrantBasePlugin implements Plugin { - - @Override - public void apply(Project project) { - project.getRootProject().getPluginManager().apply(VagrantSetupCheckerPlugin.class); - project.getRootProject().getPluginManager().apply(VagrantManagerPlugin.class); - project.getRootProject().getPluginManager().apply(ReaperPlugin.class); - - ReaperService reaper = project.getRootProject().getExtensions().getByType(ReaperService.class); - VagrantExtension extension = project.getExtensions().create("vagrant", VagrantExtension.class, project); - VagrantMachine service = project.getExtensions().create("vagrantService", VagrantMachine.class, project, extension, reaper); - - project.getGradle() - .getTaskGraph() - .whenReady( - graph -> service.refs = graph.getAllTasks() - .stream() - .filter(t -> t instanceof VagrantShellTask) - .filter(t -> t.getProject() == project) - .count() - ); - } - - /** - * Check vagrant and virtualbox versions, if any vagrant test tasks will be run. - */ - static class VagrantSetupCheckerPlugin implements Plugin { - - private static final Pattern VAGRANT_VERSION = Pattern.compile("Vagrant (\\d+\\.\\d+\\.\\d+)"); - private static final Pattern VIRTUAL_BOX_VERSION = Pattern.compile("(\\d+\\.\\d+)"); - - private final ExecOperations execOperations; - - @Inject - public VagrantSetupCheckerPlugin(ExecOperations execOperations) { - this.execOperations = execOperations; - } - - @Override - public void apply(Project project) { - if (project != project.getRootProject()) { - throw new IllegalArgumentException("VagrantSetupCheckerPlugin can only be applied to the root project of a build"); - } - - project.getGradle().getTaskGraph().whenReady(graph -> { - boolean needsVagrant = graph.getAllTasks().stream().anyMatch(t -> t instanceof VagrantShellTask); - if (needsVagrant) { - checkVersion(project, "vagrant", VAGRANT_VERSION, 1, 8, 6); - checkVersion(project, "vboxmanage", VIRTUAL_BOX_VERSION, 5, 1); - } - }); - } - - void checkVersion(Project project, String tool, Pattern versionRegex, int... minVersion) { - ByteArrayOutputStream pipe = new ByteArrayOutputStream(); - execOperations.exec(spec -> { - spec.setCommandLine(tool, "--version"); - spec.setStandardOutput(pipe); - }); - String output = pipe.toString(StandardCharsets.UTF_8).trim(); - Matcher matcher = versionRegex.matcher(output); - if (matcher.find() == false) { - throw new IllegalStateException( - tool + " version output [" + output + "] did not match regex [" + versionRegex.pattern() + "]" - ); - } - - String version = matcher.group(1); - List versionParts = Stream.of(version.split("\\.")).map(Integer::parseInt).collect(Collectors.toList()); - for (int i = 0; i < minVersion.length; ++i) { - int found = versionParts.get(i); - if (found > minVersion[i]) { - break; // most significant version is good - } else if (found < minVersion[i]) { - String exceptionMessage = String.format( - "Unsupported version of %s. Found [%s], expected [%s+", - tool, - version, - Stream.of(minVersion).map(String::valueOf).collect(Collectors.joining(".")) - ); - - throw new IllegalStateException(exceptionMessage); - } // else equal, so check next element - } - } - } - - /** - * Adds global hooks to manage destroying, starting and updating VMs. - */ - static class VagrantManagerPlugin implements Plugin, TaskActionListener, TaskExecutionListener { - - @Override - public void apply(Project project) { - if (project != project.getRootProject()) { - throw new IllegalArgumentException("VagrantManagerPlugin can only be applied to the root project of a build"); - } - project.getGradle().addListener(this); - } - - private void callIfVagrantTask(Task task, Consumer method) { - if (task instanceof VagrantShellTask) { - VagrantMachine service = task.getProject().getExtensions().getByType(VagrantMachine.class); - method.accept(service); - } - } - - @Override - public void beforeExecute(Task task) { /* nothing to do */} - - @Override - public void afterActions(Task task) { /* nothing to do */ } - - @Override - public void beforeActions(Task task) { - callIfVagrantTask(task, VagrantMachine::maybeStartVM); - } - - @Override - public void afterExecute(Task task, TaskState state) { - callIfVagrantTask(task, service -> service.maybeStopVM(state.getFailure() != null)); - } - } - -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantExtension.java b/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantExtension.java deleted file mode 100644 index 5f1b20f9eb917..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantExtension.java +++ /dev/null @@ -1,106 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.vagrant; - -import org.gradle.api.Project; -import org.gradle.api.file.RegularFileProperty; -import org.gradle.api.provider.MapProperty; -import org.gradle.api.provider.Property; -import org.gradle.api.tasks.Input; - -import java.io.File; -import java.util.Map; - -public class VagrantExtension { - - private final Property box; - private final MapProperty hostEnv; - private final MapProperty vmEnv; - private final RegularFileProperty vagrantfile; - private boolean isWindowsVM; - - public VagrantExtension(Project project) { - this.box = project.getObjects().property(String.class); - this.hostEnv = project.getObjects().mapProperty(String.class, Object.class); - this.vmEnv = project.getObjects().mapProperty(String.class, Object.class); - this.vagrantfile = project.getObjects().fileProperty(); - this.vagrantfile.convention(project.getRootProject().getLayout().getProjectDirectory().file("Vagrantfile")); - this.isWindowsVM = false; - } - - @Input - public String getBox() { - return box.get(); - } - - public void setBox(String box) { - // TODO: should verify this against the Vagrantfile, but would need to do so in afterEvaluate once vagrantfile is unmodifiable - this.box.set(box); - } - - @Input - public Map getHostEnv() { - return hostEnv.get(); - } - - public void hostEnv(String name, Object value) { - hostEnv.put(name, value); - } - - @Input - public Map getVmEnv() { - return vmEnv.get(); - } - - public void vmEnv(String name, Object value) { - vmEnv.put(name, value); - } - - @Input - public boolean isWindowsVM() { - return isWindowsVM; - } - - public void setIsWindowsVM(boolean isWindowsVM) { - this.isWindowsVM = isWindowsVM; - } - - @Input - public File getVagrantfile() { - return this.vagrantfile.get().getAsFile(); - } - - public void setVagrantfile(File file) { - vagrantfile.set(file); - } -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantMachine.java b/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantMachine.java deleted file mode 100644 index 7abf9bf5fbef6..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantMachine.java +++ /dev/null @@ -1,222 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.vagrant; - -import org.apache.commons.io.output.TeeOutputStream; -import org.opensearch.gradle.LoggedExec; -import org.opensearch.gradle.LoggingOutputStream; -import org.opensearch.gradle.ReaperService; -import org.opensearch.gradle.util.Util; -import org.gradle.api.Action; -import org.gradle.api.Project; -import org.gradle.internal.logging.progress.ProgressLogger; -import org.gradle.internal.logging.progress.ProgressLoggerFactory; - -import javax.inject.Inject; - -import java.io.File; -import java.io.OutputStream; -import java.nio.file.Paths; -import java.util.Arrays; -import java.util.Objects; -import java.util.function.UnaryOperator; - -/** - * An helper to manage a vagrant box. - *

- * This is created alongside a {@link VagrantExtension} for a project to manage starting and - * stopping a single vagrant box. - */ -public class VagrantMachine { - - private final Project project; - private final VagrantExtension extension; - private final ReaperService reaper; - // pkg private so plugin can set this after construction - long refs; - private boolean isVMStarted = false; - - public VagrantMachine(Project project, VagrantExtension extension, ReaperService reaper) { - this.project = project; - this.extension = extension; - this.reaper = reaper; - } - - @Inject - protected ProgressLoggerFactory getProgressLoggerFactory() { - throw new UnsupportedOperationException(); - } - - public void execute(Action action) { - VagrantExecSpec vagrantSpec = new VagrantExecSpec(); - action.execute(vagrantSpec); - - Objects.requireNonNull(vagrantSpec.command); - - LoggedExec.exec(project, execSpec -> { - execSpec.setExecutable("vagrant"); - File vagrantfile = extension.getVagrantfile(); - execSpec.setEnvironment(System.getenv()); // pass through env - execSpec.environment("VAGRANT_CWD", vagrantfile.getParentFile().toString()); - execSpec.environment("VAGRANT_VAGRANTFILE", vagrantfile.getName()); - extension.getHostEnv().forEach(execSpec::environment); - - execSpec.args(vagrantSpec.command); - if (vagrantSpec.subcommand != null) { - execSpec.args(vagrantSpec.subcommand); - } - execSpec.args(extension.getBox()); - if (vagrantSpec.args != null) { - execSpec.args(Arrays.asList(vagrantSpec.args)); - } - - UnaryOperator progressHandler = vagrantSpec.progressHandler; - if (progressHandler == null) { - progressHandler = new VagrantProgressLogger("==> " + extension.getBox() + ": "); - } - OutputStream output = execSpec.getStandardOutput(); - // output from vagrant needs to be manually curated because --machine-readable isn't actually "readable" - OutputStream progressStream = new ProgressOutputStream(vagrantSpec.command, progressHandler); - execSpec.setStandardOutput(new TeeOutputStream(output, progressStream)); - }); - } - - // start the configuration VM if it hasn't been started yet - void maybeStartVM() { - if (isVMStarted) { - return; - } - - execute(spec -> { - spec.setCommand("box"); - spec.setSubcommand("update"); - }); - - // Destroying before every execution can be annoying while iterating on tests locally. Therefore, we provide a flag that defaults - // to true that can be used to control whether or not to destroy any test boxes before test execution. - boolean destroyVM = Util.getBooleanProperty("vagrant.destroy", true); - if (destroyVM) { - execute(spec -> { - spec.setCommand("destroy"); - spec.setArgs("--force"); - }); - } - - // register box to be shutdown if gradle dies - reaper.registerCommand(extension.getBox(), "vagrant", "halt", "-f", extension.getBox()); - - // We lock the provider to virtualbox because the Vagrantfile specifies lots of boxes that only work - // properly in virtualbox. Virtualbox is vagrant's default but its possible to change that default and folks do. - execute(spec -> { - spec.setCommand("up"); - spec.setArgs("--provision", "--provider", "virtualbox"); - }); - isVMStarted = true; - } - - // stops the VM if refs are down to 0, or force was called - void maybeStopVM(boolean force) { - assert refs >= 1; - this.refs--; - if ((refs == 0 || force) && isVMStarted) { - execute(spec -> spec.setCommand("halt")); - reaper.unregister(extension.getBox()); - } - } - - // convert the given path from an opensearch repo path to a VM path - public static String convertLinuxPath(Project project, String path) { - return "/opensearch/" + project.getRootDir().toPath().relativize(Paths.get(path)); - } - - public static String convertWindowsPath(Project project, String path) { - return "C:\\opensearch\\" + project.getRootDir().toPath().relativize(Paths.get(path)).toString().replace('/', '\\'); - } - - public static class VagrantExecSpec { - private String command; - private String subcommand; - private String[] args; - private UnaryOperator progressHandler; - - private VagrantExecSpec() {} - - public void setCommand(String command) { - this.command = command; - } - - public void setSubcommand(String subcommand) { - this.subcommand = subcommand; - } - - public void setArgs(String... args) { - this.args = args; - } - - /** - * A function to translate output from the vagrant command execution to the progress line. - *

- * The function takes the current line of output from vagrant, and returns a new - * progress line, or {@code null} if there is no update. - */ - public void setProgressHandler(UnaryOperator progressHandler) { - this.progressHandler = progressHandler; - } - } - - private class ProgressOutputStream extends LoggingOutputStream { - - private ProgressLogger progressLogger; - private UnaryOperator progressHandler; - - ProgressOutputStream(String command, UnaryOperator progressHandler) { - this.progressHandler = progressHandler; - this.progressLogger = getProgressLoggerFactory().newOperation("vagrant"); - progressLogger.start(extension.getBox() + "> " + command, "hello"); - } - - @Override - protected void logLine(String line) { - String progress = progressHandler.apply(line); - if (progress != null) { - progressLogger.progress(progress); - } - } - - @Override - public void close() { - progressLogger.completed(); - } - } - -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantProgressLogger.java b/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantProgressLogger.java deleted file mode 100644 index 59e59b03e85f2..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantProgressLogger.java +++ /dev/null @@ -1,80 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.vagrant; - -import java.util.function.UnaryOperator; - -public class VagrantProgressLogger implements UnaryOperator { - - private static final String HEADING_PREFIX = "==> "; - - private final String squashedPrefix; - private String lastLine = ""; - private String heading = ""; - private boolean inProgressReport = false; - - public VagrantProgressLogger(String squashedPrefix) { - this.squashedPrefix = squashedPrefix; - } - - @Override - public String apply(String line) { - if (line.startsWith("\r\u001b")) { - /* We don't want to try to be a full terminal emulator but we want to - keep the escape sequences from leaking and catch _some_ of the - meaning. */ - line = line.substring(2); - if ("[K".equals(line)) { - inProgressReport = true; - } - return null; - } - if (line.startsWith(squashedPrefix)) { - line = line.substring(squashedPrefix.length()); - inProgressReport = false; - lastLine = line; - if (line.startsWith(HEADING_PREFIX)) { - line = line.substring(HEADING_PREFIX.length()); - heading = line + " > "; - } else { - line = heading + line; - } - } else if (inProgressReport) { - inProgressReport = false; - line = lastLine + line; - } else { - return null; - } - return line; - } -} diff --git a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantShellTask.java b/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantShellTask.java deleted file mode 100644 index 665f690b8b146..0000000000000 --- a/buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantShellTask.java +++ /dev/null @@ -1,130 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * - * The OpenSearch Contributors require contributions made to - * this file be licensed under the Apache-2.0 license or a - * compatible open source license. - */ - -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/* - * Modifications Copyright OpenSearch Contributors. See - * GitHub history for details. - */ - -package org.opensearch.gradle.vagrant; - -import org.gradle.api.DefaultTask; -import org.gradle.api.Project; -import org.gradle.api.tasks.Input; -import org.gradle.api.tasks.TaskAction; - -import javax.inject.Inject; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.function.UnaryOperator; -import java.util.stream.Collectors; - -import static org.opensearch.gradle.vagrant.VagrantMachine.convertLinuxPath; -import static org.opensearch.gradle.vagrant.VagrantMachine.convertWindowsPath; - -/** - * A shell script to run within a vagrant VM. - *

- * The script is run as root within the VM. - */ -public abstract class VagrantShellTask extends DefaultTask { - - private final VagrantExtension extension; - private final VagrantMachine service; - private UnaryOperator progressHandler = UnaryOperator.identity(); - private final Project project; - - @Inject - public VagrantShellTask(Project project) { - this.project = project; - this.extension = project.getExtensions().findByType(VagrantExtension.class); - if (this.extension == null) { - throw new IllegalStateException("opensearch.vagrant-base must be applied to create " + getClass().getName()); - } - this.service = project.getExtensions().getByType(VagrantMachine.class); - } - - @Input - protected abstract List getWindowsScript(); - - @Input - protected abstract List getLinuxScript(); - - @Input - public UnaryOperator getProgressHandler() { - return progressHandler; - } - - public void setProgressHandler(UnaryOperator progressHandler) { - this.progressHandler = progressHandler; - } - - @TaskAction - public void runScript() { - String rootDir = project.getRootDir().toString(); - if (extension.isWindowsVM()) { - service.execute(spec -> { - spec.setCommand("winrm"); - - List script = new ArrayList<>(); - script.add("try {"); - script.add("cd " + convertWindowsPath(project, rootDir)); - extension.getVmEnv().forEach((k, v) -> script.add("$Env:" + k + " = \"" + v + "\"")); - script.addAll(getWindowsScript().stream().map(s -> " " + s).collect(Collectors.toList())); - script.addAll( - Arrays.asList( - " exit $LASTEXITCODE", - "} catch {", - // catch if we have a failure to even run the script at all above, equivalent to set -e, sort of - " echo $_.Exception.Message", - " exit 1", - "}" - ) - ); - spec.setArgs("--elevated", "--command", String.join("\n", script)); - spec.setProgressHandler(progressHandler); - }); - } else { - service.execute(spec -> { - spec.setCommand("ssh"); - - List script = new ArrayList<>(); - script.add("sudo bash -c '"); // start inline bash script - script.add("pwd"); - script.add("cd " + convertLinuxPath(project, rootDir)); - extension.getVmEnv().forEach((k, v) -> script.add("export " + k + "=" + v)); - script.addAll(getLinuxScript()); - script.add("'"); // end inline bash script - spec.setArgs("--command", String.join("\n", script)); - spec.setProgressHandler(progressHandler); - }); - } - } - -} diff --git a/buildSrc/src/main/resources/META-INF/gradle-plugins/opensearch.distro-test.properties b/buildSrc/src/main/resources/META-INF/gradle-plugins/opensearch.distro-test.properties deleted file mode 100644 index 04ea82588f7ab..0000000000000 --- a/buildSrc/src/main/resources/META-INF/gradle-plugins/opensearch.distro-test.properties +++ /dev/null @@ -1,31 +0,0 @@ -# -# SPDX-License-Identifier: Apache-2.0 -# -# The OpenSearch Contributors require contributions made to -# this file be licensed under the Apache-2.0 license or a -# compatible open source license. -# -# Modifications Copyright OpenSearch Contributors. See -# GitHub history for details. -# - -# -# Licensed to Elasticsearch under one or more contributor -# license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright -# ownership. Elasticsearch licenses this file to you under -# the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -implementation-class=org.opensearch.gradle.test.DistroTestPlugin diff --git a/qa/os/README.md b/qa/os/README.md deleted file mode 100644 index a9b7ee269c4f2..0000000000000 --- a/qa/os/README.md +++ /dev/null @@ -1,107 +0,0 @@ -# packaging tests - -This project contains tests that verify the distributions we build work -correctly on the operating systems we support. They're intended to cover the -steps a user would take when installing and configuring an OpenSearch -distribution. They're not intended to have significant coverage of the behavior -of OpenSearch's features. - -There are two types of tests in this project. The old tests live in -`src/test/` and are written in [Bats](https://github.com/sstephenson/bats), -which is a flavor of bash scripts that run as unit tests. These tests are -deprecated because Bats is unmaintained and cannot run on Windows. - -The new tests live in `src/main/` and are written in Java. Like the old tests, -this project's tests are run inside the VM, not on your host. All new packaging -tests should be added to this set of tests if possible. - -## Running these tests - -See the section in [TESTING.md](../../TESTING.md#testing-packaging) - -## Adding a new test class - -When gradle runs the packaging tests on a VM, it runs the full suite by -default. To add a test class to the suite, add its `class` to the -`@SuiteClasses` annotation in [PackageTests.java](src/test/java/org/opensearch/packaging/test/PackageTests.java). -If a test class is added to the project but not to this annotation, it will not -run in CI jobs. The test classes are run in the order they are listed in the -annotation. - -## Choosing which distributions to test - -Distributions are represented by [enum values](src/test/java/org/opensearch/packaging/util/Distribution.java) -which know if they are compatible with the platform the tests are currently -running on. To skip a test if the distribution it's using isn't compatible with -the current platform, put this [assumption](https://github.com/junit-team/junit4/wiki/assumptions-with-assume) -in your test method or in a `@Before` method - -```java -assumeTrue(distribution.packaging.compatible); -``` - -Similarly if you write a test that is intended only for particular platforms, -you can make an assumption using the constants and methods in [Platforms.java](src/test/java/org/opensearch/packaging/util/Platforms.java) - -```java -assumeTrue("only run on windows", Platforms.WINDOWS); - -assumeTrue("only run if using systemd", Platforms.isSystemd()); -``` - -## Writing a test that covers multiple distributions - -It seems like the way to do this that makes it the most straightforward to run -and reproduce specific test cases is to create a test case class with an -abstract method that provides the distribution - -```java -public abstract class MyTestCase { - @Test - public void myTest() { /* do something with the value of #distribution() */ } - abstract Distribution distribution(); -} -``` - -and then for each distribution you want to test, create a subclass - -```java -public class MyTestDefaultTar extends MyTestCase { - @Override - Distribution distribution() { return Distribution.DEFAULT_TAR; } -} -``` - -That way when a test fails the user gets told explicitly that `MyTestDefaultTar` -failed, and to reproduce it they should run that class. See [ArchiveTests](src/test/java/org/opensearch/packaging/test/ArchiveTests.java) -and its children for an example of this. - -## Running external commands - -In general it's probably best to avoid running external commands when a good -Java alternative exists. For example most filesystem operations can be done with -the java.nio.file APIs. For those that aren't, use an instance of [Shell](src/test/java/org/opensearch/packaging/util/Shell.java) - -This class runs scripts in either bash with the `bash -c