jpabloae.blog

Release Engineering at Openbravo

Posts Tagged ‘testing

On demand branch testing for Openbravo developers

These have been some hard working but passionate weeks for Ken, an Openbravo ERP Core developer. He’s been working on implementing a couple of features in the accounting engine. In fact he’s pretty excited:

– “Wow, this is really going to be valuable for our users!”.

But hold on Ken, you think you’ve finished the development process but you haven’t. Let’s go back to the beginning, so that you can understand.

Once upon a time Ken and his team detected the need to developing a new feature. So being this a relevant change in the Core of the product, Ken decided to start developing this new feature in a new Mercurial branch. It’s the common sense choice for these cases, because he doesn’t want to integrate these changes into Core until it’s ready for general usage. Then he dived into the interesting part: he spent 2 weeks coding and writing some new tests for this new feature. So there he is, he wants the end users to enjoy this new development as soon as possible. This is legitimate and desirable, but Core has a golden rule which cannot be broken:

Anything pushed into the Core repository (erp/devel/pi) must meet a certain quality defined by our automatic tests. This is what we usually call the integration process, or simply int.

To see this simpler: all the balls in int must always be green. You might be wondering which tests they cover. Click on any of the jobs listed in the integration tab to see a description of what they do.

So Ken is now somehow a bit frightened, because although he has written quality code, and quoting his own words:

You never know for sure if you pass the tests until you pass them

And running the tests in your own machine is not an option, because from one side it’s difficult to set them up and from another side we’re continuously adding new tests to this process. As this is a legitimate concern, we’ve decided to solve this gap: we’ll provide Ken a way of running all the tests in a simulated environment, a real clone of the integration process.

So how can Ken run all the tests on his branch? Simple, just running this command:

hg push -f https://code.openbravo.com/erp/devel/try

And then monitor the jobs in the try tab. As of now only one developer can use try at a time. So you need to click on the Changes link of the active job to see if it’s testing your branch or not.

To make things easier to remember, you can add the following snippet to your main Mercurial configuration file (e.g. $HOME/.hgrc):

[paths]
try = https://code.openbravo.com/erp/devel/try

So that if you ever need to run the tests in a branch, the process would be as simple as running this command:

hg push -f try

But Ken is confused:

– “How can everyone push to the same repository? We are using different branches. Isn’t this crazy?”.

The key lies on the -f argument we’re passing to the hg push command. If you’re interested in the basic internals of how this works, every time you push your commits you’ll create a new head in the repository, and the last commit of this head will effectively become the tip of the entire repository. And the try tests are run against this tip. Confusing? If you want to have a better understanding I suggest you to play with a local repository and see how it evolves as you push new heads. If you don’t care, just use it and forget about the internals.

So what’s next? This is just the beginning of this try feature, you can consult the reference documentation. Some future plans in our TODO list:

  • Add e-mail notifications when your build starts, ends and with the final result.
  • Increase the number of developers that can use this simultaneously.

This feature is available for all the Core contributors (Openbravo S.L.U staff and external contributors). Would you like to become a Core contributor? Great! Read the following guide.

I’m sure you have suggestions about this, so we’ll be happy to hear about them. You can do it either by adding a comment here or by joining us in the #openbravo IRC channel.

I would like to thank the Mozilla team for the original idea behind this concept.

Written by jpabloae

01/07/2010 at 14:00

RM updates: ERP 2.3x discontinued, graphical upgrade test, Issue Tracker and QA integration

These are the latest news from the Release Management Team:

2.3x discontinued

Important note! The 2.3x version of Openbravo ERP has been discontinued:

  • We have removed all the continuous builds involving 2.3x from our build farm.
  • It’s no longer possible to report bugs against this version in our issue tracker.
  • The wiki documents related to 2.3x have been marked as deprecated.
  • The Mercurial code repository is permanently frozen.
  • The 2.3x installers have been removed from the SourceForge download area.

I would like remind you that if you’re running a Community Edition installation you should always be in the latest major release. Currently this means 2.50. Running an old major version is possible of course, but the voluntary support you get in the forums is provided on the latest major release. The professional support is offered in all the major releases: 2.3x, 2.40, 2.50 until now and 2.40, 2.50 after this 2.3x end of life.

So this means that if you’re running 2.3x you should seriously consider upgrading to 2.50 as soon as possible. If you have problems during this process you can either post your question in the forums, which is run by volunteers, or contact your closest partner to get professional support.

Continuous Integration: graphical upgrade test

We have a new job in our build farm that really makes a difference: perform an upgrade using the Module Management Console, starting from the last release Maintenance Pack and using to the latest daily Core OBX module. This test is now a prerequisite before merging changesets from pi to main.

It’s been developed using Selenium by the QA Team (thanks for that!) and we have adapted and integrated it into our continuous integration framework.

New issue tracker statuses

Soon we’ll have 2 new statuses in our issue tracker: Ready for integration and Ready for QA. They will be placed before the Resolved status. With this change we increase our compromise with the quality of the product. Because this means that we won’t consider an issue to be resolved until our Continuous Integration and QA Team have tested them. Check the rationale behind this decision or check the new workflow in the following graph:

Issue tracker and build farm integration

Every time a changeset is successfully promoted to erp/devel/main we automatically generate an Core OBX file out of it. Now we have integrated the issue tracker with our build farm so that a note is added with information regarding when this promotion happened, in what changeset and how to get the generated OBX file. See issue #11470 to see this in action.

For a complete list of the on-going stories we’ve been working on, check the Sprint 29 page of our Scrum spreadsheet.

Written by jpabloae

16/12/2009 at 21:02

RM updates: upgrade automation, documentation, appliance security

Following with our round of updates, here’s the latest news from the Release Management Team:

  • Continuous Integration: there is a new job that tests upgrading from the last stable MP to the latest daily OBX file. This has been done using the command line. A graphical test is in progress.
  • Live builds: the main page has been refactored and it now includes detailed build information as well as the runtime Tomcat log.
  • Appliance security updates: previous versions of the appliance are vulnerable to a man-in-the-middle attack during TLS session renegotiation. This vulnerability has been addressed in this update. Check the full changelog at the newly created appliance release notes.
  • Documentation: check the new document explaining our stack configuration in the appliances.
  • Issue Tracker: we are working on upgrading to version 1.2.0, which includes new interesting features. Many things have changed in this version so this process will take longer than a regular update. A new testing server will be announced soon.
  • Download area: the SourceForge download area has been updated to include only the latest 3 releases. The older ones have been moved to the 09-openbravo-old-releases directory.

For a complete list of the on-going stories we’ve been working on, check the Sprint 28 page of our Scrum spreadsheet.

Written by jpabloae

03/12/2009 at 09:27

ERP 2.50: experimenting with PostgreSQL 8.4 and OpenJDK 6

Currently Openbravo ERP 2.50 officially supports PostgreSQL 8.3 and Sun JDK 6. By the time when these versions were taken as our base, PostgreSQL 8.4 did not exist and OpenJDK 6 was an on-going project still not ready. One year later the situation is quite different:

  • PostgreSQL 8.4.1 is the latest and greatest of the PostgreSQL releases.
  • OpenJDK 6 is completed and a very real alternative to Sun’s JDK.

For those unfamiliar with OpenJDK, here’s a bit of history: on 2006 Sun decided to open their JDK and license it under the GPL. Some work was required for that, though: about 4% of the code was proprietary, dependent on 3rd parties who didn’t want to open those components. So the OpenJDK project started rewriting those parts and getting the rest ready. Now OpenJDK 6 has passed the Technology Compatibility Kit tests and claims to be a fully compatible Java 6 implementation.

Using OpenJDK will allow us to have a 100% open source and free software stack. Also,  as most modern Linux distributions include it as the default JDK, setting up Openbravo ERP will be easier. For example Ubuntu ships OpenJDK as the default one and in the next 10.04 LTS Sun’s JDK will be available in the multiverse repository or in none at all.

On the other hand supporting PostgreSQL 8.4 has the obvious benefits of enjoying the improvements of this new major version.

So let’s play a bit with them to see how they behave with our latest release, Openbravo ERP 2.50MP8.

PostgreSQL 8.4

I’ve run two tests using version 8.4.1:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): no surprises, our code builds cleanly with this new major version. The build times are similar compared to PostgreSQL 8.3.8.
  • Functional test (smoke test): this is usually a bit more tricky. But good news! No problems at all, it passes all the smoke test cleanly! Nice.

OpenJDK 6

Now the hard part. Let’s see how it goes:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): ouch, it fails the first time when it minifies the JavaScript files using YUI Compressor. But no worries, it’s a known issue that has a simple workaround, acceptable for now. After deleting the conflictive file the build finishes successfully. So the first big test passed, this looks promising.
  • Functional test (smoke test): so I set up Tomcat to run with OpenJDK, start the smoke tests on Openbravo ERP 2.50MP8 and 100% successfully completed! Yes, all of them.

Honestly, I’m quite impressed. I expected PostgreSQL 8.4 to work well, but I didn’t have as much faith on OpenJDK as my fellow Sree had. Congratulations to the OpenJDK team, you’ve done a great job.

Continuous Integration

You want proofs of all this? We’ve set up continuous builds and functional tests of our bleeding edge ERP code. Have a look at the experimental jobs in our build farm. This will help us detecting mismatches between our current stack and the new candidates.

The full builds will be run every 3 days and the smoke tests once per week. You can also check the trend and health of these builds.

Conclusions

My initial conclusions are clearly positive: the core of Openbravo ERP works well with PostgreSQL 8.4 and OpenJDK 6. It’s too early to say when we will officially support them, but this is an important milestone. The continuous builds and tests, as well as some manual QA and developer work will help us on taking the decision.

Written by jpabloae

05/11/2009 at 18:02

Ideas for Openbravo ERP development builds live testing

Many of our developers and power users are frequently requesting live installations where to test daily or even hourly builds of the development repositorios (pi, main, etc). We already had this internally in the headquarters of Pamplona (the so called tecnicia14 machine) and it was pretty popular among the local developers and even the consultants. In this post I collect some ideas and ask for feedback.

I’ve been thinking on why this is useful, who the target users are, how this fits with our current continuous integration framework (builds.openbravo.com), what the frequency should be, etc.

To begin with, adding this feature is on the second milestone in the Openbravo Continous Integration project.

Target audience

Firstly, I basically see three groups of users interested in this:

  1. Developers and power users: it’s obviously interesting for them to test the latest versions with different databases and running on different environments.
  2. Sporadic users who want to test the latest developments and see how the progress is going.
  3. QA team: they need these environments all the time, all the day.

The QA team has one big restriction: they need to be the only ones accessing the ERP at that time. And vice versa, developers don’t want to share the environment with QA: imagine them testing the DeleteClient functionality.

So the proposal is to have two different environments:

  • live.builds.openbravo.com: for developers, power users and sporadic curious users. Open to the world.
  • liveqa.builds.openbravo.com: for the QA team. Restricted access.

Candidate jobs

Given that we use Hudson as our CI framework, these would naturally be two different Hudson nodes. Having a look at the list of builds I’ve detected the following candidates:

The incremental jobs are only useful for testing the upgrade process, but not for live testing. That’s why they’re not included here.

Implementation

  • Every time one of the selected builds is successful we could create a tarball with a database dump and the WAR file, placing them in a known location. The sources could also be shipped, because some of the modularity features require this.
  • Then, the every new Hudson job would take their correspondent files and reset the environments using this database dump and WAR file.

As the reset process would take 5 minutes at most, the frequency could be 1-4 times a day.

Resources

Three machines would be required for this:

  • One for Oracle (shared). Because a bug in the ERP shouldn’t affect the daily builds.
  • One for Tomcat and PostgreSQL (live.builds.openbravo.com).
  • One for Tomcat and PostgreSQL (liveqa.builds.openbravo.com).

Open questions

Anything is open to questions, of course, but this is a list of topics I have brainstormed:

  • Is there any other build you would include?
  • Would you find useful to test the ERP in other operating systems and architectures other than Linux and x86/x86_64?
  • What reset frequency would be reasonable?
  • If QA needs a separate environment, do developers need it too?

What else other than listed above would be of interest for you in these live testing machines?

Written by jpabloae

01/10/2009 at 19:37

Posted in openbravo

Tagged with , ,