Development and release processes ================================= Definitions ----------- * *PyGranta meta-package*: A package which depends on the *idiomatic* packages and *auto-generated packages*. Includes documentation describing the PyGranta ecosystem, including this document. * *Idiomatic package*: A hand-written package. Typically depends on one *auto-generated package* and ``openapi-common``. There are currently three *idiomatic packages* in PyGranta: * ``ansys-grantami-bomanalytics`` * ``ansys-grantami-recordlists`` * ``ansys-grantami-jobqueue`` * *Auto-generated package*: A package generated by the *codegen template*. There are currently two *auto-generated* packages in PyGranta: * ``ansys-grantami-bomanalytics-openapi`` * ``ansys-grantami-serverapi-openapi`` * *Codegen template*: The template used to generate the *auto-generated packages*. * *Production test VM*: Virtual machine used for CI integration tests and for executing notebooks as part of the documentation build. This machine always runs the Granta MI version required to build the most recently released ``release`` branch for the *PyGranta meta-package* and all *idiomatic packages*. * *Development test VM*: Virtual machine used for CI integration tests and for executing notebooks as part of the documentation build. This machine is upgraded as required to support CI on *idiomatic package* ``main`` branches during development. Development and release timeline -------------------------------- The timeline is split into three distinct phases: 1. Phase 1: Development. There is active development upstream on closed-source Granta MI components, the *codegen template*, and on the open source *idiomatic packages* and *auto-generated packages*. Effort is made to ensure CI is passing on all ``main`` branches, but from time to time CI may fail. The *development test VM* is upgraded as required to support CI passing on ``main`` branches. 2. Phase 2: Hardening. Bugs are fixed and minor improvements are made to *idiomatic packages*, *auto-generated packages*, and the *codegen template*. During this phase the *codegen template* and *auto-generated packages* are finalized. 3. Phase 3: Pre-release. The upstream closed-source development has completed. The *PyGranta meta-package* and *idiomatic packages* are manually tested and finalized. Phase 1: Development ~~~~~~~~~~~~~~~~~~~~ During development, the following processes happen continuously: 1. Incorporate changes to upstream API definitions into *auto-generated package* repositories. 2. Check that code generation is successful by monitoring CI in the *auto-generated package* repositories. 3. If code generation is not successful, resolve by either fixing the *codegen template*, fixing the *auto-generated package* configuration, or raise the issue with the internal Ansys development team. If *idiomatic package* development does not depend on upstream API development, then there is no dependency between the *idiomatic package* and the *auto-generated package* and work can proceed in parallel. If significant *idiomatic package* development is being done that depends on the current state of an *auto-generated package*, you should update the ``main`` branch of the corresponding *idiomatic package* to depend on the *auto-generated package* ``main`` directly. During this phase, the *development test VM* may be upgraded if required to support new upstream Granta MI capabilities. Phase 2: Hardening ~~~~~~~~~~~~~~~~~~ Phase 2a: Initial dev releases ****************************** #. Update the *development test VM* with the latest Granta MI software and databases. See the ``ansys-grantami-bomanalytics`` repository for details about preparing the Restricted Substances test database. #. Publish *auto-generated packages* to the internal PyPI mirror with ``.dev`` version numbers. #. Update *idiomatic package* ``main`` branches to depend on the *auto-generated package* ``.dev`` releases. If the ``main`` branch has not been updated to depend on the *auto-generated package* ``main`` branch during development, then there may be integration issues discovered at this point - these should be resolved in the PR before merging. The PR typically includes: * Updates to Python version support to align with changes to the *auto-generated package*. * Updates to the ``README.rst`` to reflect any changes in the minimum required Granta MI version. * Changes at the interface with the *auto-generated package* if either the *codegen template* or API definition has changed. * Changes to expected test results. This is typically relevant for ``ansys-grantami-bomanalytics`` which includes acceptance-style tests. #. Publish *idiomatic packages* to the internal PyPI mirror with ``.dev`` version numbers. .. note:: At this point release branches should **not** be created. ``.dev`` releases should be published to the internal PyPI mirror from the ``main`` branch. .. note:: These ``.dev`` releases can be used for internal testing of closed-source packages that have runtime dependencies on PyGranta packages. #. Continue to review any changes to API definitions via *auto-generated packages* throughout the hardening phase. If there are any issues with the API definitions, these must be escalated with high priority. Phase 2b: Documentation review ****************************** Review the documentation for all *idiomatic packages* and the *PyGranta meta-package*: * Significant new documentation is reviewed by the appropriate reviews for linguistic and technical accuracy. * References to dependencies are updated, including: * Server software versions * Required Python versions * Compatible third-party library versions * References to Ansys documentation of other packages/products are checked to see if they need to be updated to the previous version of that documentation. .. note:: The Ansys documentation for the current in-development release is published after the PyGranta packages are released. The documentation for the previous release should be used to ensure links in published PyGranta documentation can always be resolved. Phase 2c: Release candidates **************************** Towards the end of the hardening phase, once the API definitions are determined to be stable, the *auto-generated packages* are released: #. Create ``release`` branches for *auto-generated packages* and publish as release candidates to public PyPI. #. Create Pull Requests in *idiomatic package* repositories to update ``main`` to depend on the *auto-generated package* release candidate releases. Ensure CI passes for *idiomatic packages*. #. Publish *auto-generated packages* as stable releases to public PyPI. #. Create Pull Requests in *idiomatic package* repositories to update ``main`` to depend on the *auto-generated package* stable releases. Ensure CI passes for *idiomatic packages*. #. Create ``release`` branches for *idiomatic packages* and publish as release candidates to public PyPI. #. Update the *PyGranta meta-package* ``main`` branch to depend on the public *idiomatic packages* release candidates and *auto-generated packages* stable releases. .. note:: At this point release branches should be created for *idiomatic packages*. However, during hardening these release branches still run CI against the *development test VM*. .. note:: *Idiomatic package* release candidates must depend on stable release versions of *auto-generated packages* only. They must not depend on pre-release versions of *auto-generated packages*. .. note:: When performing pre-releases of packages with towncrier, the Ansys action creates a Pull Request to update CHANGELOG.md on ``main``, and commits the CHANGELOG.md changes directly to the release branch. The commit to the release branch should be reverted, and the Pull Request to merge the changes to ``main`` should be closed and not merged. The stable *auto-generated packages* and release candidate *idiomatic packages* can now be used for the final release of internal packages. It is recommended to make any closed-source *idiomatic package* release candidate dependencies flexible enough such that they match subsequent stable releases of those *idiomatic packages*. At the end of this phase, the *PyGranta meta-package* depends on release candidates for the *idiomatic packages* and releases of the *auto-generated packages*. This guarantees that there exists a consistent set of dependencies across all *idiomatic packages* and *auto-generated packages*. Any issues at this stage should be resolved by addressing any dependency conflicts in the *idiomatic packages* and publishing new release candidates to public PyPI as necessary. Checklist ********* * |chkbx| The *development test VM* is running an up-to-date development version of Granta MI. * |chkbx| CI for all *idiomatic packages* is passing. * |chkbx| Documentation for all *idiomatic packages* and the *PyGranta meta-package* has been reviewed to verify significant new content, changes in dependencies (for example Python version support, minimum Granta MI version), and references to other Ansys documentation sources. * |chkbx| ``release`` branches have been created for *auto-generated packages* and *idiomatic packages*. * |chkbx| Stable *auto-generated packages* are available on PyPI. * |chkbx| Internal packages with runtime dependencies on *auto-generated packages* depend on the stable PyPI releases. * |chkbx| *Idiomatic packages* depend on stable releases of *auto-generated packages* only. * |chkbx| Release candidate *idiomatic packages* are available on PyPI. * |chkbx| Internal packages with runtime dependencies on *idiomatic packages* depend on the release candidate PyPI releases. * |chkbx| The *PyGranta meta-package* ``main`` branch depends on *idiomatic packages* release candidates and *auto-generated packages* stable releases only. Phase 3: Pre-release ~~~~~~~~~~~~~~~~~~~~ #. Update the *production test VM* to run the Granta MI release validated during the hardening phase. #. Update the latest *idiomatic package* ``release`` branches created during the hardening phase to run CI against the *production test VM*. #. Manually test the *PyGranta meta-package* against the Granta MI release candidate by installing the ``main`` branch directly using ``pip``. .. note:: If there have been significant changes to the *PyGranta meta-package*, a release candidate may optionally be published to PyPI. Examples of significant changes include re-organization of the documentation and changes to the CI process, both of which are not checked as part of the manual test required in this phase. Testing should occur 4 weeks following Phase 2. Any issues raised during testing should be immediately triaged and fixed or deferred. Checklist for completing this phase: * |chkbx| The *production test VM* is running the release version of Granta MI. * |chkbx| CI for all *idiomatic packages* is passing. * |chkbx| Manual testing has been completed against the *PyGranta meta-package* and all *idiomatic packages*. * |chkbx| The GitHub PyGranta Development project contains no must-deliver features or bug fixes still outstanding for the active release. Phase 4: Release ~~~~~~~~~~~~~~~~ The PyAnsys meta-package is typically released a week before Granta MI is available to customers. As a result, the stable versions of the packages should be released at least a week before Granta MI is available to customers: #. Defer any issues still associated with the active release in the GitHub PyGranta Development to a future release. #. Publish *idiomatic packages* as stable releases to public PyPI. #. Update the *PyGranta meta-package* ``main`` branch to depend on the public *idiomatic packages* stable releases. #. Create (or update if a release candidate was published in Phase 3) a ``release`` branch for the *PyGranta meta-package*. #. Publish the *PyGranta meta-package* as a stable release to public PyPI. #. Create a Pull Request in the PyAnsys meta-package repository to move the *PyGranta meta-package* dependency to the new stable release. #. If any new *idiomatic package* releases are compatible with the previous release of Granta MI and PyGranta, including the minimum supported Python version, create a patch release of the previous *PyGranta meta-package* release which includes the new *idiomatic package* releases. Checklist for completing this phase: * |chkbx| The active release in the GitHub PyGranta Development project is empty, with all deferred tickets re-assigned to a future release. * |chkbx| Stable releases of all *idiomatic packages* and the *PyGranta meta-package* are available on PyPI. * |chkbx| The PyAnsys meta-package depends on the *PyGranta meta-package* stable release. * |chkbx| A patch release has been made for the previous *PyGranta meta-package* if there is at least one backwards- compatible *idiomatic package* released during this development cycle. Addenda ------- Test VM management ~~~~~~~~~~~~~~~~~~ At all times the *production test VM* virtual machine should be running the latest validated Granta MI version. This is generally the most recent released version, but during the pre-release phase the version of Granta MI installed on the *production test VM* is more recent than the latest version available to customers. The name and URL of the *production test VM* is stored in the ``AZURE_VM_NAME`` and ``TEST_SERVER_URL`` secrets respectively. The *development test VM* runs whichever version of Granta MI is required to support CI on *idiomatic package* ``main`` branches during development. The name and URL of the *development test VM* is stored in the ``AZURE_VM_NAME_NEXT`` and ``TEST_SERVER_URL_NEXT`` secrets respectively. The *production test VM* and *development test VM* can be distinguished by their desktop backgrounds and text files in the hard drive root. This approach achieves two goals: * CI is generally passing on all *idiomatic package* ``main`` branches at any point in development and release. * The most recent *PyGranta meta-package* and all *idiomatic package* **released** ``release`` branches can still be built up to the pre-release phase to support patch releases if required. .. |chkbx| raw:: html