diff --git a/doc/release-notes/10103-rm-qa-guide.md b/doc/release-notes/10103-rm-qa-guide.md new file mode 100644 index 00000000000..1258814cd97 --- /dev/null +++ b/doc/release-notes/10103-rm-qa-guide.md @@ -0,0 +1 @@ +The QA Guide was added as an experiment in #10103 but has been removed because it has proved less useful than imagined. diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index a61a171cdf3..b2a997aa0aa 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -135,7 +135,7 @@ Conduct Performance Testing |dedicated| -See :doc:`/qa/performance-tests` for details. +See :ref:`locust` and , for example. Conduct Regression Testing --------------------------- @@ -143,8 +143,6 @@ Conduct Regression Testing |dedicated| Regression testing should be conducted on production data. -See :doc:`/qa/testing-approach` for details. -Refer to the provided regression checklist for the list of items to verify during the testing process: `Regression Checklist `_. Build the Guides for the Release Candidate ------------------------------------------ diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 4fc67583723..0ad920b0f46 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -430,8 +430,6 @@ target/coverage-it/index.html is the place to start reading the code coverage re Load/Performance Testing ------------------------ -See also :doc:`/qa/performance-tests` in the QA Guide. - .. _locust: Locust diff --git a/doc/sphinx-guides/source/index.rst b/doc/sphinx-guides/source/index.rst index 01f6b1969b5..53a12d39d93 100755 --- a/doc/sphinx-guides/source/index.rst +++ b/doc/sphinx-guides/source/index.rst @@ -25,7 +25,6 @@ These documentation guides are for the |version| version of Dataverse. To find g developers/index container/index style/index - qa/index.md How the Guides Are Organized ---------------------------- diff --git a/doc/sphinx-guides/source/qa/index.md b/doc/sphinx-guides/source/qa/index.md deleted file mode 100644 index 623b93ef31b..00000000000 --- a/doc/sphinx-guides/source/qa/index.md +++ /dev/null @@ -1,12 +0,0 @@ -# QA Guide - -```{toctree} -:caption: "Contents:" -:maxdepth: 2 -overview.md -testing-approach.md -testing-infrastructure.md -qa-workflow.md -test-automation.md -performance-tests.md -``` diff --git a/doc/sphinx-guides/source/qa/overview.md b/doc/sphinx-guides/source/qa/overview.md deleted file mode 100644 index 60e6a28ee9a..00000000000 --- a/doc/sphinx-guides/source/qa/overview.md +++ /dev/null @@ -1,63 +0,0 @@ -# Overview - -```{contents} Contents: -:local: -:depth: 3 -``` - -## Introduction - -This guide describes the testing process used by QA at IQSS and provides a reference for others filling in for that role. Please note that many variations are possible, and the main thing is to catch bugs and provide a good quality product to the user community. - -## Workflow - -Here is a brief description of our workflow: - -### Issue Submission and Prioritization: -- Members of the community or the development team submit bugs or request features through GitHub as [Issues](https://github.com/IQSS/dataverse/issues). -- These Issues are prioritized and added to a two-week-long sprint that can be tracked on the {ref}`kanban-board`. - -### Development Process: -- Developers will work on a solution on a separate branch -- Once a developer completes their work, they submit a [Pull Request](https://github.com/IQSS/dataverse/pulls) (PR). -- The PR is reviewed by a developer from the team. -- During the review, the reviewer may suggest coding or documentation changes to the original developer. - -### Quality Assurance (QA) Testing: -- The QA tester performs a smoke test of core functionality and regression testing. -- Documentation is used to understand the feature and validate any assertions made. -- If no documentation is provided in the PR, the tester may refer to the original bug report to determine the desired outcome of the changes. -- Once the branch is assumed to be safe, it is merged into the develop branch. - -### Final Steps: -- The PR and the Issue are closed and assigned the “merged” status. -- It is good practice to delete the branch if it is local. -- The content from the PR becomes part of the codebase for {doc}`future releases `. - -The complete suggested workflow can be found at {doc}`qa-workflow`. - -## Tips and Tricks - -- Start testing simply, with the most obvious test. You don’t need to know all your tests upfront. As you gain comfort and understanding of how it works, try more tests until you are done. If it is a complex feature, jot down your tests in an outline format, some beforehand as a guide, and some after as things occur to you. Save the doc in a testing folder (on Google Drive). This potentially will help with future testing. -- When in doubt, ask someone. If you are confused about how something is working, it may be something you have missed, or it could be a documentation issue, or it could be a bug! Talk to the code reviewer and the contributor/developer for their opinion and advice. -- Always tail the server.log file while testing. Open a terminal window to the test instance and `tail -F server.log`. This helps you get a real-time sense of what the server is doing when you interact with the application and makes it easier to identify any stack trace on failure. -- When overloaded, QA the simple pull requests first to reduce the queue. It gives you a mental boost to complete something and reduces the perception of the amount of work still to be done. -- When testing a bug fix, try reproducing the bug on the demo server before testing the fix. That way you know you are taking the correct steps to verify that the fix worked. -- When testing an optional feature that requires configuration, do a smoke test without the feature configured and then with it configured. That way you know that folks using the standard config are unaffected by the option if they choose not to configure it. -- Back up your DB before applying an irreversible DB update when you are using a persistent/reusable platform. Just in case it fails, and you need to carry on testing something else you can use the backup. - -## Release Cadence and Sprints - -A release likely spans multiple two-week sprints. Each sprint represents the priorities for that time and is sized so that the team can reasonably complete most of the work on time. This is a goal to help with planning, it is not a strict requirement. Some issues from the previous sprint may remain and likely be included in the next sprint but occasionally may be deprioritized and deferred to another time. - -The decision to make a release can be based on the time since the last release, some important feature needed by the community or contractual deadline, or some other logical reason to package the work completed into a named release and posted to the releases section on GitHub. - -## Test API - -The API test suite is added to and maintained by development. (See {doc}`/developers/testing` in the Developer Guide.) It is generally advisable for code contributors to add API tests when adding new functionality. The approach here is one of code coverage: exercise as much of the code base's code paths as possible, every time to catch bugs. - -This type of approach is often used to give contributing developers confidence that their code didn’t introduce any obvious, major issues and is run on each commit. Since it is a broad set of tests, it is not clear whether any specific, conceivable test is run but it does add a lot of confidence that the code base is functioning due to its reach and consistency. (See {doc}`/qa/test-automation` in the Developer Guide.) - -## Making a Release - -See {doc}`/developers/making-releases` in the Developer Guide. diff --git a/doc/sphinx-guides/source/qa/performance-tests.md b/doc/sphinx-guides/source/qa/performance-tests.md deleted file mode 100644 index 404188735a2..00000000000 --- a/doc/sphinx-guides/source/qa/performance-tests.md +++ /dev/null @@ -1,31 +0,0 @@ -# Performance Testing - -```{contents} Contents: -:local: -:depth: 3 -``` - -## Introduction - -The final testing activity before producing a release is performance testing. This could be done throughout the release cycle but since it is time-consuming, it is done once near the end. Using a load-generating tool named {ref}`Locust `, our scripts load the statistically most-loaded pages (according to Google Analytics): 50% homepage and 50% some type of dataset page. - -Since dataset page weight also varies by the number of files, a selection of about 10 datasets with varying file counts is used. The pages are called randomly as a guest user with increasing levels of user load, from 1 user to 250 users. Typical daily loads in production are around the 50-user level. Though the simulated user level does have a modest amount of random think time before repeated calls, from 5-20 seconds, it is not a real-world load so direct comparisons to production are not reliable. Instead, we compare performance to prior versions of the product, and based on how that performed in production we have some idea whether this might be similar in performance or whether there is some undetected issue that appears under load, such as inefficient or too many DB queries per page. - -## Testing Environment - -To run performance tests, we have a performance test cluster on AWS that employs web, database, and Solr. The database contains a copy of production that is updated weekly on Sundays. To ensure the homepage content is consistent between test runs across releases, two scripts set the datasets that will appear on the homepage. There is a script on the web server in the default CentOS user dir and one on the database server in the default CentOS user dir. Run these scripts before conducting the tests. - -Once the performance has been tested and recorded in a [Google spreadsheet](https://docs.google.com/spreadsheets/d/1lwPlifvgu3-X_6xLwq6Zr6sCOervr1mV_InHIWjh5KA/edit?usp=sharing) for this proposed version, the release will be prepared and posted. - -## Access - -Access to performance cluster instances requires ssh keys. The cluster itself is normally not running to reduce costs. To turn on the cluster, log on to the demo server and run the perfenv scripts from the centos default user dir. - -## Special Notes ⚠️ - -Please note the performance database is also used occasionally by members of the Curation team to generate prod reports so a courtesy check with them would be good before taking over the env. - - -Executing the Performance Script --------------------------------- -To execute the performance test script, you need to install a local copy of the database-helper-scripts project at . We have since produced a stripped-down script that calls just the collection and dataset pages and works with Python 3. diff --git a/doc/sphinx-guides/source/qa/qa-workflow.md b/doc/sphinx-guides/source/qa/qa-workflow.md deleted file mode 100644 index af462653dca..00000000000 --- a/doc/sphinx-guides/source/qa/qa-workflow.md +++ /dev/null @@ -1,104 +0,0 @@ -# QA Workflow for Pull Requests - -```{contents} Contents: -:local: -:depth: 3 -``` -## Checklist - -1. Assign the PR you are working on to yourself. - -1. What does it do? - - Read the description at the top of the PR, any release notes, documentation, and the original issue. - -1. Does it address the issue it closes? - - The PR should address the issue entirely unless otherwise noted. - -1. How do you test it? - - Look at the “how to test" section at the top of the pull request. Does it make sense? This likely won’t be the only testing you perform. You can develop further tests from the original issue or problem description, from the description of functionality, the documentation, configuration, and release notes. Also consider trying to reveal bugs by trying to break it: try bad or missing data, very large values or volume of data, exceed any place that may have a limit or boundary. - -1. Does it have or need documentation? - - Small changes or fixes usually don’t have docs but new features or extensions of a feature or new configuration options should have documentation. - -1. Does it have or need a release note snippet? - - Same as for doc, just a heads up to an admin for something of note or especially upgrade instructions as needed. See also {ref}`writing-release-note-snippets` for what to expect in a release note snippet. - -1. Does it include a database migration script (Flyway)? - - First, check the numbering in the filename of the script. It must be in line with the rules defined at {ref}`create-sql-script`. If the number is out of date (very common for older pull requests), do not merge and ask the developer to rename the script. Otherwise, deployment will fail. - - Once you're sure the numbering is ok (the next available number, basically), back up your database and proceeed with testing. - -1. Validate the documentation. - - Build the doc using Jenkins or read the automated Read the Docs preview. Does it build without errors? - Read it through for sense. - Use it for test cases and to understand the feature. - -1. Build and deploy the pull request. - - Normally this is done using Jenkins and automatically deployed to the QA test machine. See {ref}`deploy-to-internal`. - -1. Configure if required - - If needed to operate and everyone installing or upgrading will use this, configure now as all testing will use it. - -1. Smoke test the branch. - - Standard, minimal test of core functionality. - -1. Regression test-related or potentially affected features - - If config is optional and testing without config turned on, do some spot checks/ regression tests of related or potentially affected areas. - -1. Configure if optional - - What is the default, enabled or disabled? Is that clearly indicated? Test both. - By config here we mean enabling the functionality versus choosing a particular config option. Some complex features have config options in addition to enabling. Those will also need to be tested. - -1. Test all the new or changed functionality. - - The heart of the PR, what is this PR adding or fixing? Is it all there and working? - -1. Regression test related or potentially affected features. - - Sometimes new stuff modifies and extends other functionality or functionality that is shared with other aspects of the system, e.g. export, import. Check the underlying functionality that was also modified but in a spot check or briefer manner. - -1. Report any issues found within the PR - - It can be easy to lose track of what you’ve found, steps to reproduce, and any errors or stack traces from the server log. Add these in a numbered list to a comment in the pr. Easier to check off when fixed and to work on. Add large amounts of text as in the server log as attached, meaningfully named files. - -1. Retest all fixes, spot check feature functionality, smoke test - - Similar to your initial testing, it is only narrower. - -1. Test upgrade instructions, if required - - Some features build upon the existing architecture but require modifications, such as adding a new column to the DB or changing or adding data. It is crucial that this works properly for our 100+ installations. This testing should be performed at the least on the prior version with basic data objects (collection, dataset, files) and any other data that will be updated by this feature. Using the sample data from the prior version would be good or deploying to dataverse-internal and upgrading there would be a good test. Remember to back up your DB before doing a transformative upgrade so that you can repeat it later if you find a bug. - -1. Make sure the API tests in the PR have been completed and passed. - - They are run with each commit to the PR and take approximately 42 minutes to run. - -1. Merge PR - - Click the "Merge pull request" button and be sure to use the "Create a merge commit" option to include this PR into the common develop branch. - - Some of the reasons why we encourage using this option over Rebase or Squash are: - - - Preservation of commit history - - Clearer context and treaceability - - Easier collaboration, bug tracking and reverting - -1. Delete merged branch - - Just a housekeeping move if the PR is from IQSS. Click the delete branch button where the merge button had been. There is no deletion for outside contributions. - -1. Ensure that deployment to beta.dataverse.org succeeded. - - Go to to keep any eye on the deployment to to make sure it succeeded. The latest commit will appear at the bottom right and . diff --git a/doc/sphinx-guides/source/qa/test-automation.md b/doc/sphinx-guides/source/qa/test-automation.md deleted file mode 100644 index a93cb7078c1..00000000000 --- a/doc/sphinx-guides/source/qa/test-automation.md +++ /dev/null @@ -1,62 +0,0 @@ -# Test Automation -```{contents} Contents: -:local: -:depth: 3 -``` - -## Jenkins - -Jenkins is our primary tool for knowing if our API tests are passing. (Unit tests are executed locally by developers.) - -You can find our Jenkins installation at . - -Please note that while it has been open to the public in the past, it is currently firewalled off. We can poke a hole in the firewall for your IP address if necessary. Please get in touch. (You might also be interested in which is about restoring the ability of contributors to see if their pull requests are passing API tests or not.) - -### Jenkins Jobs - -Jenkins is organized into jobs. We'll highlight a few. - -#### IQSS-dataverse-develop - -, which we will refer to as the "develop" job, runs after pull requests are merged. It is crucial that this job stays green (passing) because we always want to stay in a "release ready" state. If you notice that this job is failing, make noise about it! - -You can access this job from the README at . - -#### IQSS-Dataverse-Develop-PR - - can be thought of as "PR jobs". It's a collection of jobs run on pull requests. Typically, you will navigate directly into the job (and it's particular build number) from a pull request. For example, from , look for a check called "continuous-integration/jenkins/pr-merge". Clicking it will bring you to a particular build like (build #10). - -#### guides.dataverse.org - - is what we use to build guides. See {ref}`build-guides` in the Developer Guide for how this job is used at release time. - -### Checking if API Tests are Passing on Jenkins - -If API tests are failing, you should not merge the pull request. - -How can you know if API tests are passing? Here are the steps, by way of example. - -- From the pull request, navigate to the build. For example from , look for a check called "continuous-integration/jenkins/pr-merge". Clicking it will bring you to a particular build like (build #10). -- You are now on the new "blue" interface for Jenkins. Click the button with an arrow on the right side of the header called "go to classic" which should take you to (for example) . -- Click "Test Result". -- Under "All Tests", look at the duration for "edu.harvard.iq.dataverse.api". It should be ten minutes or higher. If it was only a few seconds, tests did not run. -- Assuming tests ran, if there were failures, they should appear at the top under "All Failed Tests". Inform the author of the pull request about the error. - -### Diagnosing Failures on Jenkins - -API test failures can have multiple causes. As described above, from the "Test Result" page, you might see the failure under "All Failed Tests". However, the test could have failed because of some underlying system issue. - -If you have determined that the API tests have not run at all, your next step should be to click on "Console Output". For example, . Click "Full log" to see the full log in the browser or navigate to (for example) to get a plain text version. - -Go to the end of the log and then scroll up, looking for the failure. A failed Ansible task can look like this: - -``` -TASK [dataverse : download payara zip] ***************************************** -fatal: [localhost]: FAILED! => {"changed": false, "dest": "/tmp/payara.zip", "elapsed": 10, "msg": "Request failed: ", "url": "https://nexus.payara.fish/repository/payara-community/fish/payara/distributions/payara/VERSION/payara-VERSION.zip"} -``` - -In the example above, if Payara can't be downloaded, we're obviously going to have problems deploying Dataverse to it! - -## GitHub Actions - -We also use GitHub Actions. See for a list of actions. diff --git a/doc/sphinx-guides/source/qa/testing-approach.md b/doc/sphinx-guides/source/qa/testing-approach.md deleted file mode 100644 index 49b9075cf7f..00000000000 --- a/doc/sphinx-guides/source/qa/testing-approach.md +++ /dev/null @@ -1,50 +0,0 @@ -# Testing Approach - -```{contents} Contents: -:local: -:depth: 3 -``` -## Introduction - -We use a risk-based, manual testing approach to achieve the most benefit with limited resources. This means we want to catch bugs where they are likely to exist, ensure core functions work, and failures do not have catastrophic results. In practice this means we do a brief positive check of core functions on each build called a smoke test, we test the most likely place for new bugs to exist, the area where things have changed, and attempt to prevent catastrophic failure by asking about the scope and reach of the code and how failures may occur. - -If it seems possible through user error or some other occurrence that such a serious failure will occur, we try to make it happen in the test environment. If the code has a UI component, we also do a limited amount of browser compatibility testing using Chrome, Firefox, and Safari browsers. We do not currently do UX or accessibility testing on a regular basis, though both have been done product-wide by a Design group (in the past) and by the community. - -## Examining a Pull Request for Test Cases - -### What Problem Does It Solve? - -Read the top part of the pull request for a description, notes for reviewers, and usually a "how to test" section. Does it make sense? If not, read the underlying issue it closes and any release notes or documentation. Knowing in general what it does helps you to think about how to approach it. - -### How is It Configured? - -Most pull requests do not have any special configuration and are enabled on deployment, but some do. Configuration is part of testing. A sysadmin or superuser will need to follow these instructions so make sure they are in the release note snippet and try them out. Plus, that is the only way you will get it working to test it! - -Identify test cases by examining the problem report or feature description and any documentation of functionality. Look for statements or assertions about functions, what it does, as well as conditions or conditional behavior. These become your test cases. Think about how someone might make a mistake using it and try it. Does it fail gracefully or in a confusing, or worse, damaging manner? Also, consider whether this pull request may interact with other functionality and try some spot checks there. For instance, if new metadata fields have been added, try the export feature. Of course, try the suggestions under "how to test." Those may be sufficient, but you should always think about the pull request based on what it does. - -Try adding, modifying, and deleting any objects involved. This is probably covered by using the feature, but this is a good basic approach to keep in mind. - -Make sure any server logging is appropriate. You should tail the server log while running your tests. Watch for unreported errors or stack traces especially chatty logging. If you do find a bug you will need to report the stack trace from the server.log. Err on the side of providing the developer too much of server.log rather than too little. - -Exercise the UI if there is one. We tend to use Chrome for most of our basic testing as it's used twice as much as the next most commonly-used browser, according to our site's Google Analytics. First go through all the options in the UI. Then, if all works, spot-check using Firefox and Safari. - -Check permissions. Is this feature limited to a specific set of users? Can it be accessed by a guest or by a non-privileged user? How about pasting a privileged page URL into a non-privileged user’s browser? - -Think about risk. Is the feature or function part of a critical area such as permissions? Does the functionality modify data? You may do more testing when the risk is higher. - -## Smoke Test - -1. Go to the homepage on (this server has production data). Scroll to the bottom to ensure the build number is the one you intend to test from Jenkins. -1. Create a new user: It's fine to use a formulaic name with your initials and date and make the username and password the same, eg. kc080622. -1. Create a dataverse: You can use the same username. -1. Create a dataset: You can use the same username; fill in the required fields (do not use a template). -1. Upload 3 different types of files: You can use a tabular file, 50by1000.dta, an image file, and a text file. -1. Publish the dataset. -1. Download a file. - - -## Alternative Deployment and Testing - -This workflow is fine for a single person testing a PR, one at a time. It would be awkward or impossible if there were multiple people wanting to test different PRs at the same time. If a developer is testing, they would likely just deploy to their dev environment. That might be ok, but is the env is fully configured enough to offer a real-world testing scenario? - -An alternative might be to spin an EC2 branch on AWS, potentially using sample data. This can take some time so another option might be to spin up a few, persistent AWS instances with sample data this way, one per tester, and just deploy new builds there when you want to test. You could even configure Jenkins projects for each if desired to maintain consistency in how they’re built. diff --git a/doc/sphinx-guides/source/qa/testing-infrastructure.md b/doc/sphinx-guides/source/qa/testing-infrastructure.md deleted file mode 100644 index f07238c6b1a..00000000000 --- a/doc/sphinx-guides/source/qa/testing-infrastructure.md +++ /dev/null @@ -1,48 +0,0 @@ -# Infrastructure for Testing - -```{contents} Contents: -:local: -:depth: 3 -``` - -## Dataverse Internal - -To build and test a PR, we use a job called `IQSS_Dataverse_Internal` on (see {doc}`test-automation`), which deploys the .war file to an AWS instance named . - -(deploy-to-internal)= -### Building and Deploying a Pull Request from Jenkins to Dataverse-Internal - -1. Go to the QA column on our [project board](https://github.com/orgs/IQSS/projects/34), and select a pull request to test. - -1. From the pull request page, click the copy icon next to the pull request branch name. - -1. Log on to , select the `IQSS_Dataverse_Internal` project, and configure the repository URL and branch specifier to match the ones from the pull request. For example: - - * 8372-gdcc-xoai-library has IQSS implied - - **Repository URL:** https://github.com/IQSS/dataverse.git - - **Branch specifier:** */8372-gdcc-xoai-library - * GlobalDataverseCommunityConsortium:GDCC/DC-3B - - **Repository URL:** https://github.com/GlobalDataverseCommunityConsortium/dataverse.git - - **Branch specifier:** */GDCC/DC-3B. - -1. Click "Build Now" and note the build number in progress. - -1. Once complete, go to and check that the deployment succeeded, and that the homepage displays the latest build number. - -1. If for some reason it didn't deploy, check the server.log file. It may just be a caching issue so try un-deploying, deleting cache, restarting, and re-deploying on the server (`su - dataverse` then `/usr/local/payara7/bin/asadmin list-applications; /usr/local/payara7/bin/asadmin undeploy dataverse-6.1; /usr/local/payara7/bin/asadmin deploy /tmp/dataverse-6.1.war`) - -1. When a Jenkins job fails after a release, it might be due to the version number in the `pom.xml` file not being updated in the pull request (PR). To verify this, open the relevant GitHub issue, navigate to the PR branch, and go to `dataverse > modules > dataverse-parent > pom.xml`. Look for the version number, typically shown as `6.3`, and ensure it matches the current Dataverse build version. If it doesn't match, ask the developer to update the branch with the latest from the "develop" branch. - -1. If that didn't work, you may have run into a Flyway DB script collision error but that should be indicated by the server.log. See {doc}`/developers/sql-upgrade-scripts` in the Developer Guide. In the case of a collision, ask the developer to rename the script. - -1. Assuming the above steps worked, and they should 99% of the time, test away! Note: be sure to `tail -F server.log` in a terminal window while you are doing any testing. This way you can spot problems that may not appear in the UI and have easier access to any stack traces for easier reporting. - -## Guides Server - -There is also a guides job called `guides.dataverse.org` (see {doc}`test-automation`). Any test builds of guides are deployed to a named directory on guides.dataverse.org and can be found and tested by going to the existing guides, removing the part of the URL that contains the version, and browsing the resulting directory listing for the latest change. - -Note that changes to guides can also be previewed on Read the Docs. In the pull request, look for a link like . This Read the Docs preview is also mentioned under also {doc}`/contributor/documentation`. - -## Other Servers - -We can spin up additional AWS EC2 instances as needed. See {doc}`/developers/deployment` in the Developer Guide for the scripts we use.