What is the recommended workflow using Liquibase and Git?

Question

Recently we started using Liquibase. It didn't occurred yet but we imagined what would happen if two developers commits changes in the change log file to the shared Git repository.

How to solve or avoid a merge conflict? To broaden this question some what:

What is the recommended workflow using Liquibase in combination with Git?

Example scenario:
- Michael changes a column in table 'customer'.
- Jacob changes a column in table 'account'.
So both developers added a <changeSet> to the same changelog file changelog.xml.

EDIT :

As commented the scenario isn't that much exciting indeed. Assume Jacob was the last one to push his code. He has to pull first. Gets a warning there are merge conflicts to solve. He solves the conflict by keeping both parts of the code, that of Michael's and his. Updating the database with Liquibase gives no problems.

Advanced example scenario:
-Michael changes the name of column 'name' of table 'customer' in 'first_name', commits and pushes.
-Jacob changes the name of column 'name' of table 'customer' in 'last_name' and commits.
-Jacob gets a merge conflict when pulling Michael's code.
-Jacob and Michael discussed the conflict and agreed it has to be 'last_name', which Jacob commits and pushes.
-Michael pulls the solved conflict and runs a Liquibase update. He gets an error: column "name" does not exist

Answer 1

At my company, the way we use liquibase prevents these situations from occurring. Basically, you create a separate liquibase file for each change. We name the files after the JIRA ticket that originated the change with a little descriptive text. Each of these files, we put in a folder for the version of the system they are for; if the next release is 1.22 then that folder is created when we start making database changes and we put each liquibase file in there along with an update.xml script that just includes them. That update.xml file winds up being the only place where conflicts can really happen, and they're trivial to resolve.

To illustrate, this is the src/main/liquibase folder:

├── install                        
│   ├── projectauthor.xml          
│   ├── project_obspriorities.xml  
│   ├── project_priorities.xml     
│   ├── project_udv.xml            
│   ├── project.xml                
│   ├── roles.xml                  
│   ├── scan.xml                   
│   ├── (the other table definitions in the system go here)
│
├── install.xml                 <-- this reads all the files in ./install
│
├── local.properties            <--
├── prod.properties             <--  these are database credentials (boo, hiss)  
├── staging.properties          <-- 
├── test.properties             <--  
│
├── update.xml                  <-- reads each version/master.xml file     
│
├── v1.16
│   ├── 2013-06-06_EVL-2240.xml
│   ├── 2013-07-01_EVL-2286-remove-invalid-name-characters.xml
│   ├── 2013-07-02_defer-coauthor-projectauthor-unique-constraint.xml
│   └── master.xml
├── v1.17
│   ├── 2013-07-19_EVL-2295.xml
│   ├── 2013-09-11_EVL-2370_otf-mosaicking.xml
│   └── master.xml
├── v1.18
│   ├── 2014-05-05_EVL-2326-remove-prerequisite-construct.xml
│   ├── 2014-06-03_EVL-2750_fix-p-band-polarizations.xml
│   └── master.xml

The install.xml file is just a bunch of file inclusions:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">

    <include file="src/main/liquibase/project/install/projectauthor.xml"/>
    <include file="src/main/liquibase/project/install/project_obspriorities.xml"/>
    ...
</databaseChangeLog>

The update.xml file is the same story:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
    <include file="src/main/liquibase/project/v1.18/master.xml"/>
</databaseChangeLog>

The one aspect of the workflow I am not in love with is that the install/*.xml are supposed to create the database as it is right before the current version, but we usually don't remember to do that.

Anyway, this approach will save you from a lot of grief with merging. We're using Subversion and not having any merge difficulties with this approach.

Answer 2

There are always edge cases that need to be manually handled, but they generally happen very infrequently. Git generally handles the merging of changes at the text level just fine, so the merged file will have both changeSets in it, one after the other.

Since liquibase tracks changeSets by id/author/filename, the fact that Jacob's changeSet happens to end up before Michaels' in the final changeSet doesn't matter. When both devs run the final changeSet, Liquibase will run the other dev's changeSet only because theirs has been marked as ran but the other has not. For all other environments, both changeSets will run.

Your advanced case run into problems because both developers are making changes that are contradictory to each other. You could also run into similar problems if both developers drop a column, or add a new column with the same name. It's also not always simply one developer vs. another, sometimes conflicting changeSets come from two separate feature branches being merged in. There is no problem physically with the merged changeSet itself, the problem is that the new changelog is not logically correct. It's not really a git problem, it's a logic problem.

In practice, this type conflict happens rarely because different developers and different branches are usually working on separate areas of the codebase and when there is potential for conflict, they handle it through communication and planning.

If you do run into a conflict, there are several ways to resolve it. Usually that is handled (like in your example) by deleting incorrect or duplicate changeSets but can also be handled by creating a brand new changeSet that is a combination of both. In either case, you need to handle databases that have ran the "wrong" changeSet. How to best handle that depends on how many systems have ran it.

If it is a single developer, it is sometimes easiest to simply run liquibase changeLogSync to mark the new changeSet as ran and manually make the change in the database. If the bad changeSet was ran lately, they could even run liquibase rollbackCount X to revert their bad change and then remove the changeSet and then liquibase update

If there were multiple conflicts and/or multiple systems that have ran problem changeSets, the easiest approach is usually to use <preConditions onFail="MARK_RAN"><changeSetExecuted id=....></preConditions> tags. You can remove the bad changeSet and add a new changeSet that only runs if the old changeSet was executed and puts the database back in the state expected by later changeSets. In your example, it would rename first_name back to name so that the name to last_name changeSet works just fine.

Answer 3

I don't know liquidbase, but I think the error is at the step "Jacob and Michael discussed the conflict and agreed it has to be 'last_name', which Jacob commits and pushes."

The changelog has to match the operations to be done (or that have been done on the database). i.e. you can't remove pushed changes.

Therefore the correct resolution is name->first_name AND first_name->last_name.

Answer 4

TL;DR: Add /path/to/changelogfile.xml merge=union into .gitattributes file.

I've seen some discussions on having separate files. There are two ways of having separate files in liquibase:

1. Use the includeAll tag and let keep pushing new files to the folder.
2. Use the include tag for each separate file in the master file to dictate the order.

Both ways will still cause problems: (1) includeAll will just pick files on lexicographic order of file names, which works fine if changelog is executed every time there a new change is submitted, but this will fail if you want to deploy to a brand new file. Unless you have a naming strategy to dictate the order. But this will still be a problem in case of Michael and Jacob working at the same time: they will probably use the same name, which results in conflicts. (2) Both Michael and Jacob need to edit the last few lines of the master file. In other words: merge conflict!

The actual solution is not to change your liquibase strategy. Just keep the same strategy you've been following so far. The root of the problem here is the source control and merging mechanism. A liquibase changelog file is essentially a history. Think of it as a release notes file where developers just keep adding their work to the last line of the file. In that case, doesn't matter if two developers add two lines add the end, both should go regardless of the order. So, the solution is to create a .gitattributes file in the root folder of your git project (if you don't already have one) and add the following line to it:

/path/to/changelogfile.xml merge=union

This is still not fail-prof. To address your advanced example scenario, I can't think of any automatic strategy or workflow. The developers should communicate when they are touching the same table (or same db objects). This scenario is not only a "file" conflict, but also a semantic conflict. That's when a good CI comes handy. The developer who commits and merge the code last will get the error, and needs to get in touch with the one who won the race at first.

Note, maybe your Git portal does not support merge strategy and you still get merge conflicts on pull requests (aka merge requests). But still, when you see that and try to merge back or rebase to resolve conflicts, the conflict is already solved. I know GitLab does support it. Not sure about Github or BitBucket.