jpabloae.blog

Release Engineering at Openbravo

Posts Tagged ‘Release process

Openbravo releases: life cycle and support plan

Let’s play a game: find a word that describes in the best possible manner how your ideal ERP should be. Think about it for a minute. Many of you have probably answered Stable. I mean, once you implement it in a customer you want it to behave as you designed it to be. No more, no less. No inconveniences, no surprises.  That’s stability, right? Admittedly this sounds pretty good.

Some of you, on the other hand, might have answered Featureful, Useful, Pleasant or even Gorgeous. But hey, hold on a minute… if you want something to be Stable it can’t possibly be Beautiful, Pleasant and Featureful at the same  time. You know, business software is boring by design. That’s how it’s been up to now and don’t expect it to change in the future. Learn to live with that.

Really? Is this true? How can we find a balance between offering a user a stable but featureful, pleasant and beautiful product? Imagine an ERP that you enjoy using. Is that even possible? We believe it is, and this is why we have redesigned our releases life cycle and support plan. We started applying it in June 2010, now let me explain you how it works.

Principles:

  • We care about the ERP’s stability: a major release should be there for a long time.
  • We care about eliminating headaches to the system integrator: upgrades should be easy. And don’t force me to upgrade to the next major version, let me skip one if I want to.
  • I want a featureful ERP: one that helps me get the job done in a pleasant manner. Allow me to get the latest features in an easy manner.

Based on these principles, we have defined three phases for every major release:

  1. Preventive Support:
    • Customers receive support according to SLA.
    • Customers are entitled to report defects and expect resolution.
    • Openbravo ships regular maintenance packs.

    This phase starts on the release day and ends 6 months after the second to next release, with a minimum duration of 2.5 years.

    Example: 2.40 was released on November 2008. And 3.0 will be released on March 2011. Therefore the Preventive Support phase for 2.40 lasts till September 2011 (March 2010 + 6 months).

  2. Reactive Support – phase 1:
    • Customers receive support according to SLA.
    • Customers are entitled to report defects and expect resolution (major severity only).

    This phase lasts for 1 year.

    Example: 2.40 finishes the Preventive Support phase in September 2011. And the Reactive support – phase 1 lasts till October 2012.

  3. Reactive Support – phase 2:
    • Customers receive support according to reduced SLA (no critical tickets accepted).

    This phase lasts for 2 years.

    Example: 2.40 finishes the Reactive Support – phase 1 in September 2012. And the Reactive support – phase 2 lasts till September 2014.

Let’s illustrate this so that we can get the idea visually:

Note that Openbravo 3.1 and 3.2 are fictional releases, they are not planned nor developed yet. They are listed here to illustrate the support plans for 2.50 and 3.0 if the future 3.1 and 3.2 were to be released in 03/2012 and 03/2013 respectively.

As you can see in the graph the minimum support length of a release is of 5.5 years. Feel free to check the complete document in our wiki for more details.

We’ve seen there are different user profiles: those more conservative who look for a “Stable” product above all. And those somewhat more aggressive who look for new features and additional stimulus. Which profile fits better to you? The good news is we have a plan for both.

Advertisements

Written by jpabloae

29/11/2010 at 16:01

Posted in openbravo

Tagged with , ,

We’re branching the Openbravo 3.0 code line

Up to now the 2.50 and 3.0 versions of Openbravo have shared the same Core. “Core” is what we call the most essential module of Openbravo, the one that includes standard functionality like Financial or Sales Management.

If 2.50 and 3.0 have shared the same Core, what’s the difference between them? While a default 2.50 installation includes just Core, the 3.0 version includes a distribution of several modules: right now “Core” plus 16 more.

In terms of source code management we decided to keep one code line as much time as possible, simply because it reduces our developer’s backporting work. Backporting basically means that a bug fixed in 3.0 should also be fixed in 2.50 (if it affects 2.50, of course).

The news is that we have now reached the point where the 2.50 and 3.0 Core cannot share the same code any more. So we are planning to branch the 2.50 and 3.0 code-lines around the 15th of November.

And that’s not all about it. According to our plans we’ll be releasing version 3.0RC3 precisely the 15th of November. The extra news is that we’ll be having two 2.50 MPs on November, 2.50MP24 and 2.50MP25. Why? Simple, as the “Core” is still shared between 2.50 and 3.0, version 3.0RC3 will be needing some code not ready yet in 2.50MP23. And this is why we need 2.50MP24.

In terms of timing, we’ll freeze 2.50MP24 the 1st of November and release it the 15th of that month. So are we releasing both 2.50MP24 and 3.0RC3 the 15th of November? That’s right, and starting from that point the 2.50 and 3.0 Core code-lines will follow their own paths.

Lastly a couple of notes about how this affects to the 2.50 to 3.0 upgrade process:

  1. While we are introducing a separate branch to simplify our code management, 3.0 is just another version of core and updating from 2.50 MPx to 3.0 is not any different than updating from 2.50 MPx to 2.50 MPy. In both cases, it is a full code replacement of core and, as long as the system has been configured using the principles of modularity and does not contain any core customization, system configurations and extensions are going to be preserved during the upgrade.
  2. We will be cleaning the 3.0 core more aggressively than we are doing with the 2.50 core. Because of that, we can expect a higher volume of API changes in the 3.0 code line for the next few months and up to general availability date. However, we will continue to be conservative in 3.0 as well and accept only low risk API changes in an effort to minimize the chances to break module dependencies.  3.0 API changes will be listed in the Release Notes and users intending to upgrade from 2.50 are encouraged to monitor those notifications.

Written by jpabloae

18/10/2010 at 17:33

Posted in openbravo

Tagged with , , , ,

Introducing maturity levels to Openbravo ERP

Stability is a keyword inherently attached to ERP systems. System integrators and end users want a system that simply offers them the capabilities they need in a pleasant manner and at any moment. In this case stability means that it always works in the way they expect it to work. Openbravo takes this challenge very seriously, as you can see in our current 2.50 MPs (Maintenance Packs) release process:

  • We run a set of automated tests on every commit, which in practice means a 24/7 job: build tests, sanity tests, upgrade tests, functional tests, etc.
  • As a general rule every commit is related to a resolved issue. The QA team, together with the development team, leads the effort of individually verifying the correct resolution of these issues.
  • The QA team performs complete set of manual tests before releasing a maintenance pack to guarantee the quality of the release.

This is the global picture of our current 2.50 MP release process, delivered at the beginning of every month. Now we would like to go beyond this by offering an additional service level in our MPs. And precisely the Life Cycle Management feature introduced in version 2.50MP20 makes this possible: whenever a released MP stays live without known issues for 40 days we’ll tag that release in a special manner. So that system integrators can choose to either use the current stable versions or the ones that have matured for 40 days.

This Life Cycle Management feature introduces the concept of Maturity level for modules, which allows Openbravo to make use of its statuses as follows:

  • Test: primarily used by the Openbravo QA team. This is the pre-release status.
  • QA Approved (old name: Controlled release): this is the current maturity level of the MPs once they are released. It means they have passed the automated tests, the issues have been individually verified and the QA team has run a comprehensive set of manual tests.
  • Confirmed Stable (old name: General availability): the module has passed 40 days in the QA Approved maturity level without any known issues.

So in practice, what does this mean for a system integrator? Simple: those looking for exactly the same maturity level of the current MPs should use the QA Approved status. And those who would like to go beyond this level should consider using the Confirmed Stable status.

Configuring this is as simple as selecting the desired setting in the Module Management Console:

Note that Confirmed Stable is the default option for all the modules. We’ll apply this policy for Core starting from 2.50MP23.

 

EDIT, 2011/07/04: the maturity levels have been renamed as follows:

  • Controlled Release → QA Approved.
  • General Availability → Confirmed Stable.

Written by jpabloae

29/09/2010 at 19:03

Posted in openbravo

Tagged with , ,

Towards a better Openbravo ERP release process

Openbravo ERP 2.50 has a time based release plan. This helps us get bug fixes and new features into our user’s hands more quickly and improve our planning process.

Up to now we’ve been releasing Maintenance Packs on a monthly basis. However we have detected some improvement areas, mainly two:

  1. We want to release like a clock:  same time, same day, every month. Everyone gets benefited by this:
    • End users appreciate knowing the release dates without even reading a release calendar. They simply know that they’ll have an update at the end of every month.
    • Our engineering team can organize more effectively: developers, QA, Support, Release Management.
  2. Developers should know what commits they push go into what release. This way they can organize their work more efficiently.

So let’s start! We already have the list of desired improvements and we know that we need to set the key dates. So first we need to identify the different kind of events involved in the development process:

  • Regular commits (e.g. bugfixes or small features).
  • Large project merges (e.g. a new large feature).
  • QA automatic tests.
  • Feature freeze for the release.
  • Release packaging.
  • QA manual tests.
  • Publish the release.

Now we need to set some dates and deadlines. The QA manual work – bug found – fix it loop lasts around 10 days. So the freeze and packaging should be before the 20th. And we want to provide a guarantee to developers, so that they know which commits go into what release. Here’s a proposal:

  • All the commits pushed before the 15th of a month will go into that month’s release. We’ll make sure all the tests are passed, working together with the developers in case there’s a failure.
  • We’ll freeze the code the 18th and start the packaging right after this moment. This means we have 3 days to make the code pass all the automatic test suite. It should be more than enough. It also means that commits pushed between the 15th and the 18th might go into that month’s release. If it passes the tests of course.
  • From that moment on QA can start the manual work. If they find issues the affected developer will work on the fix and they’ll iterate on this process till QA happy is happy with the result.
  • Afterwards the Release Management team can publish the release.

And last but not least, project merges tend to create instabilities in the code main line, compared to having no reintegrations at all. So taking this into account the project merges must be done a specific time frame: from the 22nd of one month until the 5th of the next one. So that we give it enough room (10 days) to get the code stable for the freeze time.

We can see these policies summarized in the following graph:

Openbravo ERP 2.50 release timeline

Improving a release process is usually an evolution, not a revolution. And I believe this is the first step towards better timely releases.

Written by jpabloae

31/03/2010 at 16:21

Posted in openbravo

Tagged with ,

Openbravo ERP 2.40 continuous releases

Openbravo ERP has currently two major versions: 2.50 is the latest one and it’s actively developed. 2.40 is a maintenance release for Professional Edition users, and it mainly consists of bug fixes, with an average of 20 fixes per month.

These 2.40 updates are distributed as Maintenance Packs (MPs – a collection of bug fixes), with a frequency of one month during the first year and of two months during the second one (since November 2009). Starting from 2.40MP11 we are introducing a new concept of MPs. Instead of releasing bi-monthly updates we will release them on a bi-weekly basis, tested and verified by our Continuous Integration Framework. This is the workflow that this automated process follows:

  • Mike reports an issue and it gets assigned to Ken, a Openbravo ERP developer.
  • Ken fixes the issue and pushes the commit into the 2.40 Mercurial repository.
  • Our build farm runs a set of tests on that changeset:
    • Incremental update test on PostgreSQL and Oracle. This is important to test updates to this changeset.
    • Full clean builds on PostgreSQL and Oracle. This is used to test new installations based on this new changeset.
    • Verify the database consistency. This verifies that the database model matches exactly its definition in the XML files.
    • Functional tests on PostgreSQL and Oracle. This runs a set of graphical on a web browser, powered by Selenium.
    • Other minor sanity tests, such as checking that no database data is entered using the reserved ID ranges.
  • If all the tests pass successfully, we tag and sign the release in the Mercurial repository, and later on replicate it into our Subversion mirror. If one test fails the system waits for new changesets and starts the process again.
  • Mike gets notified about the availability of a new MP that includes a fix for the issue he reported.

These kind of MPs are easily identifiable by it’s version numbering, following the 2.40MPX.Y schema. As an example, we currently have released four, called 2.40MP11.1, 2.40MP11.2, 2.40MP11.3 and 2.40MP11.4.

What does this mean for a 2.40 user?

  • We are reducing the delivery time. You get maintenance updates every two weeks.
  • You get updates of the fixes you reported faster than ever.
  • You update your 2.40 installation using the usual method (Subversion), this does not change.

Note that we are also planning to release two more “traditional” MPs during the 2.40 life cycle, that will consist of additional manual verifications. They will be called 2.40MP12 and 2.40MP13, scheduled for February and August 2010 respectively. Here’s a table that summarizes the 2.40 MP frequencies:

Version Release date
2.40MP11 November 2009 (already released)
2.40MP11.X Bi-weekly
2.40MP12 February 2010
2.40MP12.X Bi-weekly
2.40MP13 August 2010

Written by jpabloae

23/12/2009 at 16:39

RM updates: home page face-lift and automation

Continuing with our promised updates about what has been done in the Release Management team during the last 2 weeks, other than the publication of the 2.50MP7 and 2.50MP8 releases.

Homepage face-lift

We have given a face-lift to the Release Management homepage, with the following major changes:

  • Release schedules and security have now a greater weight.
  • We have updated the our subprojects (what keeps us busy).
  • The RM related documentation has been organized into 4 sections.

Automation

  • New job to automatically create the API documentation (javadoc) for new maintenance packs.
  • Huge progress towards achieving a total release automation.

For a complete list of the on-going stories we’ve been working on, check the Sprint 26 page of our Scrum spreadsheet.

Written by jpabloae

04/11/2009 at 23:58

RM updates: Continuous Integration, Ubuntu, Javadoc, Release Notes

Like in any other team, some of the tasks we do in the Release Management Team have consequences in users, other times in developers and some times even in other company staff. We feel it’s important to notify other teams about what has been done. Not the progress or the future plans, we have Scrum of Scrums for that. But about the actual specific results.

So this is a simple as this: whenever there’s something new we think it could useful to a specific groups of users we’ll write a short summary to them. This will happen every 2 weeks.

Continuous Integration

  • There’s a new set of tests called erp_sanity_tests. This intends to collect various basic checks that are not worth to keep as a single Hudson job. As for now there’s one included, to Check if the primary keys of the database in erp/devel/pi that were present as columns in erp/stable/2.40 have a onCreateDefault value set.New tests are accepted either by describing them to us or by sending us the code to be run.
  • The QuickStart and Module Installation jobs have been moved from slave1 to slave2. We currently have 4 machines for builds.openbravo.com, and with this change one 100% dedicated to smoke tests.This affect developers in the way that now smoke tests will be finished quicker without interruptions from other jobs.

Ubuntu package

There was a demand for instructions on how to install the Ubuntu package with Tomcat in one server and PostgreSQL in another. This steps are currently manual and they’ve been documented.

Release Notes

They have been refactored. Now there’s a main page where you can see all the versions.

Then you can access the specific summary notes for each release. Check 2.50MP6 for an example.

And if you’re interested in more details, the Changelog page shows the complete list of fixed issues.

The 3 pages are linked. And the idea is to replace the Changelog page at some point, by delegating in Mantis to do this automatically.

Openbravo ERP API Javadoc

The Openbravo ERP API is now available for online viewing.

This includes all the releases since 2.50, as well as docs for erp/devel/main and erp/devel/pi, which are automatically updated in each incremental build of our continous integration framework.

Written by jpabloae

15/10/2009 at 11:19