jpabloae.blog

Release Engineering at Openbravo

Posts Tagged ‘QA

Introducing maturity levels to Openbravo ERP

Stability is a keyword inherently attached to ERP systems. System integrators and end users want a system that simply offers them the capabilities they need in a pleasant manner and at any moment. In this case stability means that it always works in the way they expect it to work. Openbravo takes this challenge very seriously, as you can see in our current 2.50 MPs (Maintenance Packs) release process:

  • We run a set of automated tests on every commit, which in practice means a 24/7 job: build tests, sanity tests, upgrade tests, functional tests, etc.
  • As a general rule every commit is related to a resolved issue. The QA team, together with the development team, leads the effort of individually verifying the correct resolution of these issues.
  • The QA team performs complete set of manual tests before releasing a maintenance pack to guarantee the quality of the release.

This is the global picture of our current 2.50 MP release process, delivered at the beginning of every month. Now we would like to go beyond this by offering an additional service level in our MPs. And precisely the Life Cycle Management feature introduced in version 2.50MP20 makes this possible: whenever a released MP stays live without known issues for 40 days we’ll tag that release in a special manner. So that system integrators can choose to either use the current stable versions or the ones that have matured for 40 days.

This Life Cycle Management feature introduces the concept of Maturity level for modules, which allows Openbravo to make use of its statuses as follows:

  • Test: primarily used by the Openbravo QA team. This is the pre-release status.
  • QA Approved (old name: Controlled release): this is the current maturity level of the MPs once they are released. It means they have passed the automated tests, the issues have been individually verified and the QA team has run a comprehensive set of manual tests.
  • Confirmed Stable (old name: General availability): the module has passed 40 days in the QA Approved maturity level without any known issues.

So in practice, what does this mean for a system integrator? Simple: those looking for exactly the same maturity level of the current MPs should use the QA Approved status. And those who would like to go beyond this level should consider using the Confirmed Stable status.

Configuring this is as simple as selecting the desired setting in the Module Management Console:

Note that Confirmed Stable is the default option for all the modules. We’ll apply this policy for Core starting from 2.50MP23.

 

EDIT, 2011/07/04: the maturity levels have been renamed as follows:

  • Controlled Release → QA Approved.
  • General Availability → Confirmed Stable.

Written by jpabloae

29/09/2010 at 19:03

Posted in openbravo

Tagged with , ,

On demand branch testing for Openbravo developers

These have been some hard working but passionate weeks for Ken, an Openbravo ERP Core developer. He’s been working on implementing a couple of features in the accounting engine. In fact he’s pretty excited:

– “Wow, this is really going to be valuable for our users!”.

But hold on Ken, you think you’ve finished the development process but you haven’t. Let’s go back to the beginning, so that you can understand.

Once upon a time Ken and his team detected the need to developing a new feature. So being this a relevant change in the Core of the product, Ken decided to start developing this new feature in a new Mercurial branch. It’s the common sense choice for these cases, because he doesn’t want to integrate these changes into Core until it’s ready for general usage. Then he dived into the interesting part: he spent 2 weeks coding and writing some new tests for this new feature. So there he is, he wants the end users to enjoy this new development as soon as possible. This is legitimate and desirable, but Core has a golden rule which cannot be broken:

Anything pushed into the Core repository (erp/devel/pi) must meet a certain quality defined by our automatic tests. This is what we usually call the integration process, or simply int.

To see this simpler: all the balls in int must always be green. You might be wondering which tests they cover. Click on any of the jobs listed in the integration tab to see a description of what they do.

So Ken is now somehow a bit frightened, because although he has written quality code, and quoting his own words:

You never know for sure if you pass the tests until you pass them

And running the tests in your own machine is not an option, because from one side it’s difficult to set them up and from another side we’re continuously adding new tests to this process. As this is a legitimate concern, we’ve decided to solve this gap: we’ll provide Ken a way of running all the tests in a simulated environment, a real clone of the integration process.

So how can Ken run all the tests on his branch? Simple, just running this command:

hg push -f https://code.openbravo.com/erp/devel/try

And then monitor the jobs in the try tab. As of now only one developer can use try at a time. So you need to click on the Changes link of the active job to see if it’s testing your branch or not.

To make things easier to remember, you can add the following snippet to your main Mercurial configuration file (e.g. $HOME/.hgrc):

[paths]
try = https://code.openbravo.com/erp/devel/try

So that if you ever need to run the tests in a branch, the process would be as simple as running this command:

hg push -f try

But Ken is confused:

– “How can everyone push to the same repository? We are using different branches. Isn’t this crazy?”.

The key lies on the -f argument we’re passing to the hg push command. If you’re interested in the basic internals of how this works, every time you push your commits you’ll create a new head in the repository, and the last commit of this head will effectively become the tip of the entire repository. And the try tests are run against this tip. Confusing? If you want to have a better understanding I suggest you to play with a local repository and see how it evolves as you push new heads. If you don’t care, just use it and forget about the internals.

So what’s next? This is just the beginning of this try feature, you can consult the reference documentation. Some future plans in our TODO list:

  • Add e-mail notifications when your build starts, ends and with the final result.
  • Increase the number of developers that can use this simultaneously.

This feature is available for all the Core contributors (Openbravo S.L.U staff and external contributors). Would you like to become a Core contributor? Great! Read the following guide.

I’m sure you have suggestions about this, so we’ll be happy to hear about them. You can do it either by adding a comment here or by joining us in the #openbravo IRC channel.

I would like to thank the Mozilla team for the original idea behind this concept.

Written by jpabloae

01/07/2010 at 14:00

Openbravo ERP 2.40 continuous releases

Openbravo ERP has currently two major versions: 2.50 is the latest one and it’s actively developed. 2.40 is a maintenance release for Professional Edition users, and it mainly consists of bug fixes, with an average of 20 fixes per month.

These 2.40 updates are distributed as Maintenance Packs (MPs – a collection of bug fixes), with a frequency of one month during the first year and of two months during the second one (since November 2009). Starting from 2.40MP11 we are introducing a new concept of MPs. Instead of releasing bi-monthly updates we will release them on a bi-weekly basis, tested and verified by our Continuous Integration Framework. This is the workflow that this automated process follows:

  • Mike reports an issue and it gets assigned to Ken, a Openbravo ERP developer.
  • Ken fixes the issue and pushes the commit into the 2.40 Mercurial repository.
  • Our build farm runs a set of tests on that changeset:
    • Incremental update test on PostgreSQL and Oracle. This is important to test updates to this changeset.
    • Full clean builds on PostgreSQL and Oracle. This is used to test new installations based on this new changeset.
    • Verify the database consistency. This verifies that the database model matches exactly its definition in the XML files.
    • Functional tests on PostgreSQL and Oracle. This runs a set of graphical on a web browser, powered by Selenium.
    • Other minor sanity tests, such as checking that no database data is entered using the reserved ID ranges.
  • If all the tests pass successfully, we tag and sign the release in the Mercurial repository, and later on replicate it into our Subversion mirror. If one test fails the system waits for new changesets and starts the process again.
  • Mike gets notified about the availability of a new MP that includes a fix for the issue he reported.

These kind of MPs are easily identifiable by it’s version numbering, following the 2.40MPX.Y schema. As an example, we currently have released four, called 2.40MP11.1, 2.40MP11.2, 2.40MP11.3 and 2.40MP11.4.

What does this mean for a 2.40 user?

  • We are reducing the delivery time. You get maintenance updates every two weeks.
  • You get updates of the fixes you reported faster than ever.
  • You update your 2.40 installation using the usual method (Subversion), this does not change.

Note that we are also planning to release two more “traditional” MPs during the 2.40 life cycle, that will consist of additional manual verifications. They will be called 2.40MP12 and 2.40MP13, scheduled for February and August 2010 respectively. Here’s a table that summarizes the 2.40 MP frequencies:

Version Release date
2.40MP11 November 2009 (already released)
2.40MP11.X Bi-weekly
2.40MP12 February 2010
2.40MP12.X Bi-weekly
2.40MP13 August 2010

Written by jpabloae

23/12/2009 at 16:39

RM updates: ERP 2.3x discontinued, graphical upgrade test, Issue Tracker and QA integration

These are the latest news from the Release Management Team:

2.3x discontinued

Important note! The 2.3x version of Openbravo ERP has been discontinued:

  • We have removed all the continuous builds involving 2.3x from our build farm.
  • It’s no longer possible to report bugs against this version in our issue tracker.
  • The wiki documents related to 2.3x have been marked as deprecated.
  • The Mercurial code repository is permanently frozen.
  • The 2.3x installers have been removed from the SourceForge download area.

I would like remind you that if you’re running a Community Edition installation you should always be in the latest major release. Currently this means 2.50. Running an old major version is possible of course, but the voluntary support you get in the forums is provided on the latest major release. The professional support is offered in all the major releases: 2.3x, 2.40, 2.50 until now and 2.40, 2.50 after this 2.3x end of life.

So this means that if you’re running 2.3x you should seriously consider upgrading to 2.50 as soon as possible. If you have problems during this process you can either post your question in the forums, which is run by volunteers, or contact your closest partner to get professional support.

Continuous Integration: graphical upgrade test

We have a new job in our build farm that really makes a difference: perform an upgrade using the Module Management Console, starting from the last release Maintenance Pack and using to the latest daily Core OBX module. This test is now a prerequisite before merging changesets from pi to main.

It’s been developed using Selenium by the QA Team (thanks for that!) and we have adapted and integrated it into our continuous integration framework.

New issue tracker statuses

Soon we’ll have 2 new statuses in our issue tracker: Ready for integration and Ready for QA. They will be placed before the Resolved status. With this change we increase our compromise with the quality of the product. Because this means that we won’t consider an issue to be resolved until our Continuous Integration and QA Team have tested them. Check the rationale behind this decision or check the new workflow in the following graph:

Issue tracker and build farm integration

Every time a changeset is successfully promoted to erp/devel/main we automatically generate an Core OBX file out of it. Now we have integrated the issue tracker with our build farm so that a note is added with information regarding when this promotion happened, in what changeset and how to get the generated OBX file. See issue #11470 to see this in action.

For a complete list of the on-going stories we’ve been working on, check the Sprint 29 page of our Scrum spreadsheet.

Written by jpabloae

16/12/2009 at 21:02

Ideas for Openbravo ERP development builds live testing

Many of our developers and power users are frequently requesting live installations where to test daily or even hourly builds of the development repositorios (pi, main, etc). We already had this internally in the headquarters of Pamplona (the so called tecnicia14 machine) and it was pretty popular among the local developers and even the consultants. In this post I collect some ideas and ask for feedback.

I’ve been thinking on why this is useful, who the target users are, how this fits with our current continuous integration framework (builds.openbravo.com), what the frequency should be, etc.

To begin with, adding this feature is on the second milestone in the Openbravo Continous Integration project.

Target audience

Firstly, I basically see three groups of users interested in this:

  1. Developers and power users: it’s obviously interesting for them to test the latest versions with different databases and running on different environments.
  2. Sporadic users who want to test the latest developments and see how the progress is going.
  3. QA team: they need these environments all the time, all the day.

The QA team has one big restriction: they need to be the only ones accessing the ERP at that time. And vice versa, developers don’t want to share the environment with QA: imagine them testing the DeleteClient functionality.

So the proposal is to have two different environments:

  • live.builds.openbravo.com: for developers, power users and sporadic curious users. Open to the world.
  • liveqa.builds.openbravo.com: for the QA team. Restricted access.

Candidate jobs

Given that we use Hudson as our CI framework, these would naturally be two different Hudson nodes. Having a look at the list of builds I’ve detected the following candidates:

The incremental jobs are only useful for testing the upgrade process, but not for live testing. That’s why they’re not included here.

Implementation

  • Every time one of the selected builds is successful we could create a tarball with a database dump and the WAR file, placing them in a known location. The sources could also be shipped, because some of the modularity features require this.
  • Then, the every new Hudson job would take their correspondent files and reset the environments using this database dump and WAR file.

As the reset process would take 5 minutes at most, the frequency could be 1-4 times a day.

Resources

Three machines would be required for this:

  • One for Oracle (shared). Because a bug in the ERP shouldn’t affect the daily builds.
  • One for Tomcat and PostgreSQL (live.builds.openbravo.com).
  • One for Tomcat and PostgreSQL (liveqa.builds.openbravo.com).

Open questions

Anything is open to questions, of course, but this is a list of topics I have brainstormed:

  • Is there any other build you would include?
  • Would you find useful to test the ERP in other operating systems and architectures other than Linux and x86/x86_64?
  • What reset frequency would be reasonable?
  • If QA needs a separate environment, do developers need it too?

What else other than listed above would be of interest for you in these live testing machines?

Written by jpabloae

01/10/2009 at 19:37

Posted in openbravo

Tagged with , ,