Since I published Part 2 of the Git Log Viewer, our process has been updated to include a PreProduction (PreProd) environment. We found that only having one testing environment and our extensive use of Feature Toggles meant that only the new features (or new implementations of existing functionality) were being tested and not the existing code.
However the name PreProd is a bit of a misnomer, as the environment is not an accurate mimic of our Live environment. We also have the Live Back environment (original name I know!) which is an exact copy of the production database and code.
One example of the use of our feature toggles is for the data importers, which are used to get data in and out of our product. We are in the process of re-writing them for performance reasons but because there are 15 of them, we are tackling them one at a time. We use feature toggles to enable or disable each new importer/exporter as we need to, so that we can continue to release to production without a half-completed importer being used.
As each new importer/exporter combination is completed we switch it on in the Test environment to get feedback, but on occasion we have failed to ensure that the current importer/exporter for that data set is still functional. Sometimes we would only discover this in Production and would have to deploy a hotfix. The addition of the PreProd environment aims to help resolve this issue by giving us two testing environments where we can directly compare the old and new functionality without changing any configuration, or re-deploying any code.
Our new improved and updated Change Log Viewer now looks like this:
We can now see the changes in all 4 environments in a single view. The addition of another environment was fairly straight forward, and just required us to repeat the setup we already had for the other environments. In order to show more data in the same real estate, I did have to compress the columns and tweak the padding/margins but the end result is still very readable.
You will also notice that the image shows a Master and Production branch tab. In Part 2 I explained how we migrated the binaries of the website from each environment, applying the necessary configuration changes along the way. However I did not cover how we handled hotfixes.
Our current process is as follows: We have two main branches, Master and Production. All development is done on Master and CI builds this branch on each push from a developer. Using the
build number from Jenkins, we embed the version number in the binaries in the format
following the Microsoft versioning scheme (as opposed to Semantic Versioning). After the build is completed and the integration and unit tests have passed we deploy the build to Development to run the Selenium tests.
Our usual workflow is to push all changes to Test, then PreProd and finally Production. Once we have a stable version in Production, we reset the Production branch to point to this version in the Master branch history. This allows us to have a base to work from if we need to fix a show stopper bug, or release an emergency feature aka the hotfix.
In the hotfix situation we will first make the change on the Master branch and test it per the normal workflow. Once we have feedback that the change is correct, we will cherry pick that commit (or series of commits) into the Production branch. The Production branch is then built by Jenkins which updates the
revision number of the original Master Branch version. For example if we have built
220.127.116.11 from the Master branch and we need to create a hotfix, then the Production branch will have version
18.104.22.168. We can cherry pick multiple commits into one push, and the hotfix will contain all of those changes.
Once these cherry picked commits have been pushed to the Production branch, Jenkins will rebuild the binaries and then deploy this version to our Live Back environment. Once we get sign off of this version in Live Back, it is deployed to Production.
The display of the Production branch changes is shown in the same manner as the Master branch changes:
On Jenkins, the Master and Production build pipelines now regenerate the
changelog-production.json files and the Live Back and PreProd websites have been added to the automatic version number scrapping. If the
revision number of the Production version number is non-zero, then we know that Production environment currently has a Production branch build, and we switch the source of the displayed commits behind the Production column accordingly.
At the end of the day, this workflow allows us to continue getting the benefits of Trunk Based Development, while still catering for unexpected requirements or show stopper bugs. Perhaps it can help you as well?