Release Engineering at Openbravo

Towards a better Openbravo ERP release process

Openbravo ERP 2.50 has a time based release plan. This helps us get bug fixes and new features into our user’s hands more quickly and improve our planning process.

Up to now we’ve been releasing Maintenance Packs on a monthly basis. However we have detected some improvement areas, mainly two:

  1. We want to release like a clock:  same time, same day, every month. Everyone gets benefited by this:
    • End users appreciate knowing the release dates without even reading a release calendar. They simply know that they’ll have an update at the end of every month.
    • Our engineering team can organize more effectively: developers, QA, Support, Release Management.
  2. Developers should know what commits they push go into what release. This way they can organize their work more efficiently.

So let’s start! We already have the list of desired improvements and we know that we need to set the key dates. So first we need to identify the different kind of events involved in the development process:

  • Regular commits (e.g. bugfixes or small features).
  • Large project merges (e.g. a new large feature).
  • QA automatic tests.
  • Feature freeze for the release.
  • Release packaging.
  • QA manual tests.
  • Publish the release.

Now we need to set some dates and deadlines. The QA manual work – bug found – fix it loop lasts around 10 days. So the freeze and packaging should be before the 20th. And we want to provide a guarantee to developers, so that they know which commits go into what release. Here’s a proposal:

  • All the commits pushed before the 15th of a month will go into that month’s release. We’ll make sure all the tests are passed, working together with the developers in case there’s a failure.
  • We’ll freeze the code the 18th and start the packaging right after this moment. This means we have 3 days to make the code pass all the automatic test suite. It should be more than enough. It also means that commits pushed between the 15th and the 18th might go into that month’s release. If it passes the tests of course.
  • From that moment on QA can start the manual work. If they find issues the affected developer will work on the fix and they’ll iterate on this process till QA happy is happy with the result.
  • Afterwards the Release Management team can publish the release.

And last but not least, project merges tend to create instabilities in the code main line, compared to having no reintegrations at all. So taking this into account the project merges must be done a specific time frame: from the 22nd of one month until the 5th of the next one. So that we give it enough room (10 days) to get the code stable for the freeze time.

We can see these policies summarized in the following graph:

Openbravo ERP 2.50 release timeline

Improving a release process is usually an evolution, not a revolution. And I believe this is the first step towards better timely releases.

Written by jpabloae

31/03/2010 at 16:21

Posted in openbravo

Tagged with ,

Amazon EC2: backup management tips

Cloud Computing services offer a great deal of advantages to end users and sysadmins. I’m sure most of us appreciate not taking care about the hardware, having agility to provision new resources, getting the ability to recover fast from disasters, taking benefit of a very high network bandwidth speed, and so forth. It’s a very good solution for many situations.

However managing a Cloud is different compared to the traditional server management. We have been using Amazon EC2 in Openbravo for the last couple of years, so I would like to share some tips with you with regards to backup management and recovery.

DNS management: Elastic IPs

Making backups is the first step towards a successful system recovery. But in order to take benefit of these backups it is also essential to react in a quick and easy manner when something goes wrong, so that we have the system is back up and running in a matter of minutes.  One of the biggest delays tied to the traditional server management comes with the DNS changes. That is, migrating to a new machine usually means having a new IP address. And therefore it forces you to update the DNS records, and this might take days to propagate.

Amazon has a nice feature called Elastic IP that allows you to forget about this. You basically allocate an IP address for you, so that you can assign it to any of your machines “on the fly”.

Think about the difference:

  1. Traditional DNS management: if a new server gives me a new IP I have to change the DNS records.
  2. Elastic IP DNS management: I can assign my IP address to any of my instances.

The first option might take several days to be spread in the entire world, while the second one is immediate.

EBS: persistent storage

Amazon EC2 has continuously improved the features they offer. One of the limitations it initially had consisted of the fact that the hard drives where not persistent. That is, if you shut down a machine you lose your data. In September 2008 Amazon introduced the EBS hard drives (Elastic Block Storage), which means persistent storage that essentially works like a normal hard drive that you can attach to any of your instances. However you could not run the entire operating system in one of these EBS drivers. In any case this was a big step forward, given that you could save your critical dynamic data (e.g. a database) in a persistent storage. Starting from December 2009 you can finally run an entire operating system in a EBS unit.

This feature is relatively new so most of the public AMIs are still based “instance-store”, instead of in “EBS-store”. The recommendation here is incremental:

  • Make sure you save your critical dynamic data in a EBS drive.
  • If possible, run the entire instance in a EBS drive.

I’ve just said that it works similar to a regular hard drive you can attach to any instance. This is not entirely true, it’s better: you can make snapshots of the entire disk. As many as you want, and they are incremental. You can of course restore them into a new drive any time you want.

So as a last tip with EBS do regular snapshots, and make sure you purge the old ones.

Off-site backups

There’s a golden rule with regards to backups: “Do not put all the eggs in the same basket”. It is very unlikely for Amazon EC2 to suffer a general downtime or disaster so that you would loose your data (e.g. a fire). They make sure this won’t happen and they do their backup homework as well. In any case it is generally a good idea to have a second recovery plan, physically isolated from your main backup location.

Amazon EC2 currently has two independent regions (US and EU), so the first option is to replicate the backups from region to another. However if a malicious user gets your EC2 credentials they might have temptations to wipe out all your data in both regions. To avoid this, as a recommendation, create an extra EC2/S3 account with complete read-only access to the first account. So that your backups cannot be compromised in that way.

If you are more paranoid than this, you can schedule weekly or biweekly backups to an external storage service provider.

Physical backups

It is sometimes very useful to have a recent backup available with you at your office. One option is to download it. But depending on the size of your backups and your network speed this might be prohibitive. Amazon has a nice feature called Import/Export that covers this need:

  • You send a hard drive to them with instructions.
  • They load the requested data into the hard drive.
  • They send you the hard drive back.

Openbravo ERP and EC2

OK, those tips sounds reasonable. So what should I specifically do with my Openbravo ERP instance in EC2? Stay tunned, a new post will be coming soon covering this topic.

Written by jpabloae

10/03/2010 at 16:11

Posted in openbravo

Tagged with ,

SSH: a Swiss knife for system integrators

Give me a place to stand on, and I will move the Earth.

This is a legend ascribed to the famous Archimedes, genius of antiquity. When he understood the basic principles behind the lever he felt its power and he started seeing them in a different way. His paradigm changed.

In a similar way SSH is a tool that can completely change the way you work, for good. Let me show you why you, as a system integrator, should be interested in its wonders. And I can assure you that if you get familiar with the basic principles behind it you’ll be able to perform tasks you would never possibly imagine.

Case 1: Browsing through a remote computer

There are multiple situations where browsing through a remote computer is interesting:

  • IP address restrictions: a customer has restricted access so that I can only access a remote machine from my network at work. I am at home and I really need to access this server.
  • Content filtering: I am located in a network or a country that restricts my Internet connection. And I really need to access some pages which are key to do my job.
  • Remote LAN access: I have access to a remote computer. But I would like to access the ERP or database of another computer in the same local network.

So this is the magic that makes this possible:

ssh -D 8888 johndoe@remote_computer

This basically converts the remote computer into a proxy server only for you. So that you just need to provide this information to your web browser. Using Firefox as an example, you’d need to go to Preferences → Advanced → Network → Settings, then select Manual proxy configuration and finally enter localhost in the SOCKS Host field and 8888 as the port number. The simplest way to verify it’s working is to visit so that you can verify your IP address, it should be the remote one.

Firefox Preferences, Advanced sectionFirefox proxy settings

Case 2: Securely connect to a remote database

This is a typical case scenario. The remote machine has only SSH opened and there is no direct access to the database. Let’s suppose it’s PostgreSQL running on port 5432 on the remote computer, but the port is not opened to the outside world, only to local connections. So as have SSH access you can ask it to redirect the remote 5432 port into any port of your local machine, like 5433:

ssh -L 5433:localhost:5432 johndoe@remote_computer

Now you can start psql, pgAdmin or your favorite client and use localhost as the host and 5433 as the port in the connection details.

PgAdmin3 through a SSH tunnel

As for Oracle the concept is the same and just the port numbers change:

ssh -L 1522:localhost:1521 johndoe@remote_computer

Case 3: Expose my local ERP into a remote network

Let’s suppose that I have Openbravo ERP beautifully running in my local machine, it includes some nice new changes we’ve been working on. I would like to show it to Mike and Sandra, but they are located in a remote network. I am in a hotel, there is no way I can ask the IT staff of the hotel to open a port for my users to access the ERP in my computer.  SSH comes to the rescue again: basically you can perform the opposite operation of Case 2, and forward your local Web Server port into any port of a remote machine:

ssh -R :9999:localhost:80 johndoe@remote_computer

So now I can ask Mike and Sandra to enter http://local_ip_of_remote_computer:9999 and bingo, they can access my ERP installation.

Important note: for this feature to work the server’s SSH configuration (sshd_config) must have the GatewayPorts option set to yes.

Case 4: Securely connect to a remote database available only in the LAN

Now let’s suppose I have SSH access to remote_computer, but not to remote_computer-2, which is is in the same LAN as the first one. And I want to access the database in remote_computer-2 using my graphical SQL client. There are multiple ways of solving this situation, by using variants of Case 1 or Case 2.  We’ll do it extending the first case. Firstly,  open the SSH connection and establish the local proxy server:

ssh -D 8888 johndoe@remote_computer

Now we want to tell our PostgreSQL client to use this proxy. But usually they don’t support this feature. So here proxychains comes to the rescue. This is a tool that allows you to make any program use the Internet connection through that proxy. Once it is installed, it requires a minimal configuration in $HOME/.proxychains/proxychains.conf, only required the first time you use it:

tcp_read_time_out 15000
tcp_connect_time_out 10000

socks5 8888

From now on you can prepend the proxychains command to your program and it will go to the Internet using the proxy server connection. So for example in our case we would go a terminal and run:

proxychains pgadmin3


proxychains psql -d openbravo -U tad -h localhost -p 5433


As you can see SSH opens a new world of possibilities for you. Invest some time playing with it, you won’t regret.

Some final words for Windows users: don’t worry, this is not valid for UNIX based systems only. If you run Windows in your computer you can use PuTTY to achieve exactly the same results.

UPDATE (2010/04/26): adding the GatewayPorts requirement and the corrected ssh command based on Asier’s comments.

Written by jpabloae

27/02/2010 at 12:37

Posted in openbravo

Tagged with ,

Openbravo ERP 2.40 continuous releases

Openbravo ERP has currently two major versions: 2.50 is the latest one and it’s actively developed. 2.40 is a maintenance release for Professional Edition users, and it mainly consists of bug fixes, with an average of 20 fixes per month.

These 2.40 updates are distributed as Maintenance Packs (MPs – a collection of bug fixes), with a frequency of one month during the first year and of two months during the second one (since November 2009). Starting from 2.40MP11 we are introducing a new concept of MPs. Instead of releasing bi-monthly updates we will release them on a bi-weekly basis, tested and verified by our Continuous Integration Framework. This is the workflow that this automated process follows:

  • Mike reports an issue and it gets assigned to Ken, a Openbravo ERP developer.
  • Ken fixes the issue and pushes the commit into the 2.40 Mercurial repository.
  • Our build farm runs a set of tests on that changeset:
    • Incremental update test on PostgreSQL and Oracle. This is important to test updates to this changeset.
    • Full clean builds on PostgreSQL and Oracle. This is used to test new installations based on this new changeset.
    • Verify the database consistency. This verifies that the database model matches exactly its definition in the XML files.
    • Functional tests on PostgreSQL and Oracle. This runs a set of graphical on a web browser, powered by Selenium.
    • Other minor sanity tests, such as checking that no database data is entered using the reserved ID ranges.
  • If all the tests pass successfully, we tag and sign the release in the Mercurial repository, and later on replicate it into our Subversion mirror. If one test fails the system waits for new changesets and starts the process again.
  • Mike gets notified about the availability of a new MP that includes a fix for the issue he reported.

These kind of MPs are easily identifiable by it’s version numbering, following the 2.40MPX.Y schema. As an example, we currently have released four, called 2.40MP11.1, 2.40MP11.2, 2.40MP11.3 and 2.40MP11.4.

What does this mean for a 2.40 user?

  • We are reducing the delivery time. You get maintenance updates every two weeks.
  • You get updates of the fixes you reported faster than ever.
  • You update your 2.40 installation using the usual method (Subversion), this does not change.

Note that we are also planning to release two more “traditional” MPs during the 2.40 life cycle, that will consist of additional manual verifications. They will be called 2.40MP12 and 2.40MP13, scheduled for February and August 2010 respectively. Here’s a table that summarizes the 2.40 MP frequencies:

Version Release date
2.40MP11 November 2009 (already released)
2.40MP11.X Bi-weekly
2.40MP12 February 2010
2.40MP12.X Bi-weekly
2.40MP13 August 2010

Written by jpabloae

23/12/2009 at 16:39

RM updates: ERP 2.3x discontinued, graphical upgrade test, Issue Tracker and QA integration

These are the latest news from the Release Management Team:

2.3x discontinued

Important note! The 2.3x version of Openbravo ERP has been discontinued:

  • We have removed all the continuous builds involving 2.3x from our build farm.
  • It’s no longer possible to report bugs against this version in our issue tracker.
  • The wiki documents related to 2.3x have been marked as deprecated.
  • The Mercurial code repository is permanently frozen.
  • The 2.3x installers have been removed from the SourceForge download area.

I would like remind you that if you’re running a Community Edition installation you should always be in the latest major release. Currently this means 2.50. Running an old major version is possible of course, but the voluntary support you get in the forums is provided on the latest major release. The professional support is offered in all the major releases: 2.3x, 2.40, 2.50 until now and 2.40, 2.50 after this 2.3x end of life.

So this means that if you’re running 2.3x you should seriously consider upgrading to 2.50 as soon as possible. If you have problems during this process you can either post your question in the forums, which is run by volunteers, or contact your closest partner to get professional support.

Continuous Integration: graphical upgrade test

We have a new job in our build farm that really makes a difference: perform an upgrade using the Module Management Console, starting from the last release Maintenance Pack and using to the latest daily Core OBX module. This test is now a prerequisite before merging changesets from pi to main.

It’s been developed using Selenium by the QA Team (thanks for that!) and we have adapted and integrated it into our continuous integration framework.

New issue tracker statuses

Soon we’ll have 2 new statuses in our issue tracker: Ready for integration and Ready for QA. They will be placed before the Resolved status. With this change we increase our compromise with the quality of the product. Because this means that we won’t consider an issue to be resolved until our Continuous Integration and QA Team have tested them. Check the rationale behind this decision or check the new workflow in the following graph:

Issue tracker and build farm integration

Every time a changeset is successfully promoted to erp/devel/main we automatically generate an Core OBX file out of it. Now we have integrated the issue tracker with our build farm so that a note is added with information regarding when this promotion happened, in what changeset and how to get the generated OBX file. See issue #11470 to see this in action.

For a complete list of the on-going stories we’ve been working on, check the Sprint 29 page of our Scrum spreadsheet.

Written by jpabloae

16/12/2009 at 21:02

RM updates: upgrade automation, documentation, appliance security

Following with our round of updates, here’s the latest news from the Release Management Team:

  • Continuous Integration: there is a new job that tests upgrading from the last stable MP to the latest daily OBX file. This has been done using the command line. A graphical test is in progress.
  • Live builds: the main page has been refactored and it now includes detailed build information as well as the runtime Tomcat log.
  • Appliance security updates: previous versions of the appliance are vulnerable to a man-in-the-middle attack during TLS session renegotiation. This vulnerability has been addressed in this update. Check the full changelog at the newly created appliance release notes.
  • Documentation: check the new document explaining our stack configuration in the appliances.
  • Issue Tracker: we are working on upgrading to version 1.2.0, which includes new interesting features. Many things have changed in this version so this process will take longer than a regular update. A new testing server will be announced soon.
  • Download area: the SourceForge download area has been updated to include only the latest 3 releases. The older ones have been moved to the 09-openbravo-old-releases directory.

For a complete list of the on-going stories we’ve been working on, check the Sprint 28 page of our Scrum spreadsheet.

Written by jpabloae

03/12/2009 at 09:27

ERP 2.50: experimenting with PostgreSQL 8.4 and OpenJDK 6

Currently Openbravo ERP 2.50 officially supports PostgreSQL 8.3 and Sun JDK 6. By the time when these versions were taken as our base, PostgreSQL 8.4 did not exist and OpenJDK 6 was an on-going project still not ready. One year later the situation is quite different:

  • PostgreSQL 8.4.1 is the latest and greatest of the PostgreSQL releases.
  • OpenJDK 6 is completed and a very real alternative to Sun’s JDK.

For those unfamiliar with OpenJDK, here’s a bit of history: on 2006 Sun decided to open their JDK and license it under the GPL. Some work was required for that, though: about 4% of the code was proprietary, dependent on 3rd parties who didn’t want to open those components. So the OpenJDK project started rewriting those parts and getting the rest ready. Now OpenJDK 6 has passed the Technology Compatibility Kit tests and claims to be a fully compatible Java 6 implementation.

Using OpenJDK will allow us to have a 100% open source and free software stack. Also,  as most modern Linux distributions include it as the default JDK, setting up Openbravo ERP will be easier. For example Ubuntu ships OpenJDK as the default one and in the next 10.04 LTS Sun’s JDK will be available in the multiverse repository or in none at all.

On the other hand supporting PostgreSQL 8.4 has the obvious benefits of enjoying the improvements of this new major version.

So let’s play a bit with them to see how they behave with our latest release, Openbravo ERP 2.50MP8.

PostgreSQL 8.4

I’ve run two tests using version 8.4.1:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): no surprises, our code builds cleanly with this new major version. The build times are similar compared to PostgreSQL 8.3.8.
  • Functional test (smoke test): this is usually a bit more tricky. But good news! No problems at all, it passes all the smoke test cleanly! Nice.

OpenJDK 6

Now the hard part. Let’s see how it goes:

  • Full build of Openbravo ERP 2.50MP8 (ant install.source): ouch, it fails the first time when it minifies the JavaScript files using YUI Compressor. But no worries, it’s a known issue that has a simple workaround, acceptable for now. After deleting the conflictive file the build finishes successfully. So the first big test passed, this looks promising.
  • Functional test (smoke test): so I set up Tomcat to run with OpenJDK, start the smoke tests on Openbravo ERP 2.50MP8 and 100% successfully completed! Yes, all of them.

Honestly, I’m quite impressed. I expected PostgreSQL 8.4 to work well, but I didn’t have as much faith on OpenJDK as my fellow Sree had. Congratulations to the OpenJDK team, you’ve done a great job.

Continuous Integration

You want proofs of all this? We’ve set up continuous builds and functional tests of our bleeding edge ERP code. Have a look at the experimental jobs in our build farm. This will help us detecting mismatches between our current stack and the new candidates.

The full builds will be run every 3 days and the smoke tests once per week. You can also check the trend and health of these builds.


My initial conclusions are clearly positive: the core of Openbravo ERP works well with PostgreSQL 8.4 and OpenJDK 6. It’s too early to say when we will officially support them, but this is an important milestone. The continuous builds and tests, as well as some manual QA and developer work will help us on taking the decision.

Written by jpabloae

05/11/2009 at 18:02