Platform- and software-as-a-service have fundamentally changed how software is delivered and maintained. For these services, different software works in concert to deliver a comprehensive service, including operating systems, application servers, databases, and more. The service you use often defines the “software stack” employed. Service providers take on the responsibility of keeping the software stack up-to-date, because if they don’t, they may become vulnerable to the ever-evolving threat landscape. In fact, Help Net Security stated this past October that the “majority of 2019 breaches were the result of unapplied security patches.”
The challenge facing development teams that deliver these platform and software services is the method by which they address critical bugs and vulnerabilities. There was a time when software fixes, maintenance, etc., were delivered through a hotfix process if the need fell outside of the normal release cycle. This practice was common when software was delivered on premises, or even in single-tenant cloud instances. However, with the wide adoption of multi-tenant services, hotfixes can lead to unforeseen consequences. As a result, hotfixes are becoming a thing of the past. In fact, Microsoft eliminated its hotfix service in late 2018.
What is a hotfix?
Hotfixes are a common practice in software development, where a critical bug is identified and a fix is issued outside of the typical release process. Unlike a patch, a hotfix does not go through the extensive regression testing that a patch does, as it is typically targeted at providing a fix to a single issue. This means that hotfixes can introduce unexpected issues when released to the core code, taking down applications, functions, or introducing security vulnerabilities.
Hotfix releases are dwindling because continuous delivery and integration (CI/CD) best practices are being widely adopted. These practices allow development teams to work on smaller pieces of code to make changes, improvements, and even fix bugs on a continuous release cycle. The advantage is that all the code branches are subject to rigorous unit testing, resulting in higher quality releases. As more software companies adopt this practice, developers and admins will be released from the burden of maintaining hotfixes, and that is a very good thing.
Why some software vendors still use hotfixes
The decision product managers face on whether to release a hotfix is a balance of the impact of the identified bug versus the potential to cause downstream impacts. With on-premises software, the risks introduced by a hotfix were naturally limited by the number of impacted customers. If additional issues were introduced by the hotfix, they would be addressed in the next patch, which was fully regression tested.
When you migrate to a cloud delivery model, everything is amplified. Ideally in a cloud service, every client is operating on a common release. As cloud services leverage advanced architectures like containers and hybrid tenancy, any downstream effects of the hotfix have the potential to impact more clients.
Is DevOps a Silver Bullet for Hotfixes?
Teams that operate using DevOps best practices are not forced into deploying free of hotfixes. There are plenty of available resources you can find that will detail the steps taken to set up a hotfix pipeline. A hotfix pipeline will help you speed your hotfix code branch to deployment and fix unwanted issues quickly, but they do not include the battery of regression tests that ensure the fix will seamlessly integrate with all of the software code. DevOps simply enables you to automate the hotfix process.
While it may not be a silver bullet, a strong continuous delivery and integration practice can help you to speed patches to market and avoid hotfixes altogether. This will be determined by the strategies your development teams employ.
The impact microservice architectures have on hotfixes
A microservice architecture deconstructs a system enabling it to run as independent services, rather than a traditional monolithic stack. Because each service operates independently in this type of architecture, teams execute test plans in isolation as well as part of the whole. Traditional testing has limits when you are delivering your system as a highly distributed system.
One of the key tenets of a development and testing plan for a microservice-based architecture is a trunk-based development methodology. This methodology centralizes the code into a single branch that developers collaborate on, minimizing merges at the time of release. Trunk-based releases eliminate the feature and release branches that can prove problematic when releasing patches.
Why Pega is moving away from hotfixes
Hotfixes are typically born from patching a bug, back-porting a feature change, or modifying code to meet specific client needs. While these provide a short-term remedy for a specific challenge, the process leads to selective patching and a high incidence of unique and untested code. The long-term implication of these practices leads to an uneven upgrade process. This is driven by the dearth of combinations of patches that may (or may not) have been applied throughout the software lifecycle.
Migrating to trunk-based patch releases enables development teams to eliminate hotfixes (and hotfix pipelines) by delivering patches that meet all quality gates. By doing this, development teams eliminate the risk introduced by untested combinations of code and ensure a consistent high-quality user experience. The final result supports the argument for implementing DevOps best practices, by delivering higher quality releases, faster.
Learn More:
- See how Pega’s open platform supports DevOps out of the box.
- Learn how Pega Cloud® Services support scalable, secure enterprise architectures.
- Read more from Pete Brown on the topics of evolving cloud service architecture, tenancy, data security controls, and future-proof integration strategies.
- Join us at PegaWorld iNspire to test-drive the latest game-changing software and network with tech leaders from some of the world’s biggest global brands.