jpabloae.blog

Release Engineering at Openbravo

Posts Tagged ‘Continuous Integration

On demand branch testing for Openbravo developers

These have been some hard working but passionate weeks for Ken, an Openbravo ERP Core developer. He’s been working on implementing a couple of features in the accounting engine. In fact he’s pretty excited:

– “Wow, this is really going to be valuable for our users!”.

But hold on Ken, you think you’ve finished the development process but you haven’t. Let’s go back to the beginning, so that you can understand.

Once upon a time Ken and his team detected the need to developing a new feature. So being this a relevant change in the Core of the product, Ken decided to start developing this new feature in a new Mercurial branch. It’s the common sense choice for these cases, because he doesn’t want to integrate these changes into Core until it’s ready for general usage. Then he dived into the interesting part: he spent 2 weeks coding and writing some new tests for this new feature. So there he is, he wants the end users to enjoy this new development as soon as possible. This is legitimate and desirable, but Core has a golden rule which cannot be broken:

Anything pushed into the Core repository (erp/devel/pi) must meet a certain quality defined by our automatic tests. This is what we usually call the integration process, or simply int.

To see this simpler: all the balls in int must always be green. You might be wondering which tests they cover. Click on any of the jobs listed in the integration tab to see a description of what they do.

So Ken is now somehow a bit frightened, because although he has written quality code, and quoting his own words:

You never know for sure if you pass the tests until you pass them

And running the tests in your own machine is not an option, because from one side it’s difficult to set them up and from another side we’re continuously adding new tests to this process. As this is a legitimate concern, we’ve decided to solve this gap: we’ll provide Ken a way of running all the tests in a simulated environment, a real clone of the integration process.

So how can Ken run all the tests on his branch? Simple, just running this command:

hg push -f https://code.openbravo.com/erp/devel/try

And then monitor the jobs in the try tab. As of now only one developer can use try at a time. So you need to click on the Changes link of the active job to see if it’s testing your branch or not.

To make things easier to remember, you can add the following snippet to your main Mercurial configuration file (e.g. $HOME/.hgrc):

[paths]
try = https://code.openbravo.com/erp/devel/try

So that if you ever need to run the tests in a branch, the process would be as simple as running this command:

hg push -f try

But Ken is confused:

– “How can everyone push to the same repository? We are using different branches. Isn’t this crazy?”.

The key lies on the -f argument we’re passing to the hg push command. If you’re interested in the basic internals of how this works, every time you push your commits you’ll create a new head in the repository, and the last commit of this head will effectively become the tip of the entire repository. And the try tests are run against this tip. Confusing? If you want to have a better understanding I suggest you to play with a local repository and see how it evolves as you push new heads. If you don’t care, just use it and forget about the internals.

So what’s next? This is just the beginning of this try feature, you can consult the reference documentation. Some future plans in our TODO list:

  • Add e-mail notifications when your build starts, ends and with the final result.
  • Increase the number of developers that can use this simultaneously.

This feature is available for all the Core contributors (Openbravo S.L.U staff and external contributors). Would you like to become a Core contributor? Great! Read the following guide.

I’m sure you have suggestions about this, so we’ll be happy to hear about them. You can do it either by adding a comment here or by joining us in the #openbravo IRC channel.

I would like to thank the Mozilla team for the original idea behind this concept.

Written by jpabloae

01/07/2010 at 14:00

Openbravo ERP 2.40 continuous releases

Openbravo ERP has currently two major versions: 2.50 is the latest one and it’s actively developed. 2.40 is a maintenance release for Professional Edition users, and it mainly consists of bug fixes, with an average of 20 fixes per month.

These 2.40 updates are distributed as Maintenance Packs (MPs – a collection of bug fixes), with a frequency of one month during the first year and of two months during the second one (since November 2009). Starting from 2.40MP11 we are introducing a new concept of MPs. Instead of releasing bi-monthly updates we will release them on a bi-weekly basis, tested and verified by our Continuous Integration Framework. This is the workflow that this automated process follows:

  • Mike reports an issue and it gets assigned to Ken, a Openbravo ERP developer.
  • Ken fixes the issue and pushes the commit into the 2.40 Mercurial repository.
  • Our build farm runs a set of tests on that changeset:
    • Incremental update test on PostgreSQL and Oracle. This is important to test updates to this changeset.
    • Full clean builds on PostgreSQL and Oracle. This is used to test new installations based on this new changeset.
    • Verify the database consistency. This verifies that the database model matches exactly its definition in the XML files.
    • Functional tests on PostgreSQL and Oracle. This runs a set of graphical on a web browser, powered by Selenium.
    • Other minor sanity tests, such as checking that no database data is entered using the reserved ID ranges.
  • If all the tests pass successfully, we tag and sign the release in the Mercurial repository, and later on replicate it into our Subversion mirror. If one test fails the system waits for new changesets and starts the process again.
  • Mike gets notified about the availability of a new MP that includes a fix for the issue he reported.

These kind of MPs are easily identifiable by it’s version numbering, following the 2.40MPX.Y schema. As an example, we currently have released four, called 2.40MP11.1, 2.40MP11.2, 2.40MP11.3 and 2.40MP11.4.

What does this mean for a 2.40 user?

  • We are reducing the delivery time. You get maintenance updates every two weeks.
  • You get updates of the fixes you reported faster than ever.
  • You update your 2.40 installation using the usual method (Subversion), this does not change.

Note that we are also planning to release two more “traditional” MPs during the 2.40 life cycle, that will consist of additional manual verifications. They will be called 2.40MP12 and 2.40MP13, scheduled for February and August 2010 respectively. Here’s a table that summarizes the 2.40 MP frequencies:

Version Release date
2.40MP11 November 2009 (already released)
2.40MP11.X Bi-weekly
2.40MP12 February 2010
2.40MP12.X Bi-weekly
2.40MP13 August 2010

Written by jpabloae

23/12/2009 at 16:39

RM updates: ERP 2.3x discontinued, graphical upgrade test, Issue Tracker and QA integration

These are the latest news from the Release Management Team:

2.3x discontinued

Important note! The 2.3x version of Openbravo ERP has been discontinued:

  • We have removed all the continuous builds involving 2.3x from our build farm.
  • It’s no longer possible to report bugs against this version in our issue tracker.
  • The wiki documents related to 2.3x have been marked as deprecated.
  • The Mercurial code repository is permanently frozen.
  • The 2.3x installers have been removed from the SourceForge download area.

I would like remind you that if you’re running a Community Edition installation you should always be in the latest major release. Currently this means 2.50. Running an old major version is possible of course, but the voluntary support you get in the forums is provided on the latest major release. The professional support is offered in all the major releases: 2.3x, 2.40, 2.50 until now and 2.40, 2.50 after this 2.3x end of life.

So this means that if you’re running 2.3x you should seriously consider upgrading to 2.50 as soon as possible. If you have problems during this process you can either post your question in the forums, which is run by volunteers, or contact your closest partner to get professional support.

Continuous Integration: graphical upgrade test

We have a new job in our build farm that really makes a difference: perform an upgrade using the Module Management Console, starting from the last release Maintenance Pack and using to the latest daily Core OBX module. This test is now a prerequisite before merging changesets from pi to main.

It’s been developed using Selenium by the QA Team (thanks for that!) and we have adapted and integrated it into our continuous integration framework.

New issue tracker statuses

Soon we’ll have 2 new statuses in our issue tracker: Ready for integration and Ready for QA. They will be placed before the Resolved status. With this change we increase our compromise with the quality of the product. Because this means that we won’t consider an issue to be resolved until our Continuous Integration and QA Team have tested them. Check the rationale behind this decision or check the new workflow in the following graph:

Issue tracker and build farm integration

Every time a changeset is successfully promoted to erp/devel/main we automatically generate an Core OBX file out of it. Now we have integrated the issue tracker with our build farm so that a note is added with information regarding when this promotion happened, in what changeset and how to get the generated OBX file. See issue #11470 to see this in action.

For a complete list of the on-going stories we’ve been working on, check the Sprint 29 page of our Scrum spreadsheet.

Written by jpabloae

16/12/2009 at 21:02

RM updates: upgrade automation, documentation, appliance security

Following with our round of updates, here’s the latest news from the Release Management Team:

  • Continuous Integration: there is a new job that tests upgrading from the last stable MP to the latest daily OBX file. This has been done using the command line. A graphical test is in progress.
  • Live builds: the main page has been refactored and it now includes detailed build information as well as the runtime Tomcat log.
  • Appliance security updates: previous versions of the appliance are vulnerable to a man-in-the-middle attack during TLS session renegotiation. This vulnerability has been addressed in this update. Check the full changelog at the newly created appliance release notes.
  • Documentation: check the new document explaining our stack configuration in the appliances.
  • Issue Tracker: we are working on upgrading to version 1.2.0, which includes new interesting features. Many things have changed in this version so this process will take longer than a regular update. A new testing server will be announced soon.
  • Download area: the SourceForge download area has been updated to include only the latest 3 releases. The older ones have been moved to the 09-openbravo-old-releases directory.

For a complete list of the on-going stories we’ve been working on, check the Sprint 28 page of our Scrum spreadsheet.

Written by jpabloae

03/12/2009 at 09:27

ERP 2.50: experimenting with PostgreSQL 8.4 and OpenJDK 6

Currently Openbravo ERP 2.50 officially supports PostgreSQL 8.3 and Sun JDK 6. By the time when these versions were taken as our base, PostgreSQL 8.4 did not exist and OpenJDK 6 was an on-going project still not ready. One year later the situation is quite different:

  • PostgreSQL 8.4.1 is the latest and greatest of the PostgreSQL releases.
  • OpenJDK 6 is completed and a very real alternative to Sun’s JDK.

For those unfamiliar with OpenJDK, here’s a bit of history: on 2006 Sun decided to open their JDK and license it under the GPL. Some work was required for that, though: about 4% of the code was proprietary, dependent on 3rd parties who didn’t want to open those components. So the OpenJDK project started rewriting those parts and getting the rest ready. Now OpenJDK 6 has passed the Technology Compatibility Kit tests and claims to be a fully compatible Java 6 implementation.

Using OpenJDK will allow us to have a 100% open source and free software stack. Also,  as most modern Linux distributions include it as the default JDK, setting up Openbravo ERP will be easier. For example Ubuntu ships OpenJDK as the default one and in the next 10.04 LTS Sun’s JDK will be available in the multiverse repository or in none at all.

On the other hand supporting PostgreSQL 8.4 has the obvious benefits of enjoying the improvements of this new major version.

So let’s play a bit with them to see how they behave with our latest release, Openbravo ERP 2.50MP8.

PostgreSQL 8.4

I’ve run two tests using version 8.4.1:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): no surprises, our code builds cleanly with this new major version. The build times are similar compared to PostgreSQL 8.3.8.
  • Functional test (smoke test): this is usually a bit more tricky. But good news! No problems at all, it passes all the smoke test cleanly! Nice.

OpenJDK 6

Now the hard part. Let’s see how it goes:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): ouch, it fails the first time when it minifies the JavaScript files using YUI Compressor. But no worries, it’s a known issue that has a simple workaround, acceptable for now. After deleting the conflictive file the build finishes successfully. So the first big test passed, this looks promising.
  • Functional test (smoke test): so I set up Tomcat to run with OpenJDK, start the smoke tests on Openbravo ERP 2.50MP8 and 100% successfully completed! Yes, all of them.

Honestly, I’m quite impressed. I expected PostgreSQL 8.4 to work well, but I didn’t have as much faith on OpenJDK as my fellow Sree had. Congratulations to the OpenJDK team, you’ve done a great job.

Continuous Integration

You want proofs of all this? We’ve set up continuous builds and functional tests of our bleeding edge ERP code. Have a look at the experimental jobs in our build farm. This will help us detecting mismatches between our current stack and the new candidates.

The full builds will be run every 3 days and the smoke tests once per week. You can also check the trend and health of these builds.

Conclusions

My initial conclusions are clearly positive: the core of Openbravo ERP works well with PostgreSQL 8.4 and OpenJDK 6. It’s too early to say when we will officially support them, but this is an important milestone. The continuous builds and tests, as well as some manual QA and developer work will help us on taking the decision.

Written by jpabloae

05/11/2009 at 18:02

RM updates: home page face-lift and automation

Continuing with our promised updates about what has been done in the Release Management team during the last 2 weeks, other than the publication of the 2.50MP7 and 2.50MP8 releases.

Homepage face-lift

We have given a face-lift to the Release Management homepage, with the following major changes:

  • Release schedules and security have now a greater weight.
  • We have updated the our subprojects (what keeps us busy).
  • The RM related documentation has been organized into 4 sections.

Automation

  • New job to automatically create the API documentation (javadoc) for new maintenance packs.
  • Huge progress towards achieving a total release automation.

For a complete list of the on-going stories we’ve been working on, check the Sprint 26 page of our Scrum spreadsheet.

Written by jpabloae

04/11/2009 at 23:58

RM updates: Continuous Integration, Ubuntu, Javadoc, Release Notes

Like in any other team, some of the tasks we do in the Release Management Team have consequences in users, other times in developers and some times even in other company staff. We feel it’s important to notify other teams about what has been done. Not the progress or the future plans, we have Scrum of Scrums for that. But about the actual specific results.

So this is a simple as this: whenever there’s something new we think it could useful to a specific groups of users we’ll write a short summary to them. This will happen every 2 weeks.

Continuous Integration

  • There’s a new set of tests called erp_sanity_tests. This intends to collect various basic checks that are not worth to keep as a single Hudson job. As for now there’s one included, to Check if the primary keys of the database in erp/devel/pi that were present as columns in erp/stable/2.40 have a onCreateDefault value set.New tests are accepted either by describing them to us or by sending us the code to be run.
  • The QuickStart and Module Installation jobs have been moved from slave1 to slave2. We currently have 4 machines for builds.openbravo.com, and with this change one 100% dedicated to smoke tests.This affect developers in the way that now smoke tests will be finished quicker without interruptions from other jobs.

Ubuntu package

There was a demand for instructions on how to install the Ubuntu package with Tomcat in one server and PostgreSQL in another. This steps are currently manual and they’ve been documented.

Release Notes

They have been refactored. Now there’s a main page where you can see all the versions.

Then you can access the specific summary notes for each release. Check 2.50MP6 for an example.

And if you’re interested in more details, the Changelog page shows the complete list of fixed issues.

The 3 pages are linked. And the idea is to replace the Changelog page at some point, by delegating in Mantis to do this automatically.

Openbravo ERP API Javadoc

The Openbravo ERP API is now available for online viewing.

This includes all the releases since 2.50, as well as docs for erp/devel/main and erp/devel/pi, which are automatically updated in each incremental build of our continous integration framework.

Written by jpabloae

15/10/2009 at 11:19