Refactoring Liquibase changelog files

Question

We are using Liquibase for this project I'm working on right now, and all the changelogs are in one big XML-file. Unfortunately, this file has gotten WAY too big - and we want to introduce a "master" file, that includes the original file and any new ones.

The old structure:

/db/changesets-from-beginning-of-time.xml

The new structure:

/db/changesets/changesets-from-beginning-of-time.xml
/db/changesets/changesets-v.1.2.3.xml
/db/changesets/changeset-master.xml

The content of the changesets-* -files are simply changeset xml, while the changeset-master.xml file looks like this:

<databaseChangeLog xmlns="...skipped...">
    <include file="changesets-from-beginning-of-time.xml" 
              relativeToChangelogFile="true"/>
    <include file="changesets-v1.2.3.xml" 
              relativeToChangelogFile="true"/>
</databaseChangeLog>

Now, the DATABASECHANGELOG table in my database references the old files, and thus the old changesets are run again.

According to the Liquibase documentation, each changeset is uniquely identified by the combination [filepath/-name]:::[id]:::[author] - which is less than optimal for me.

So my question is - how do I refactor the file structure without breaking my liquibase setup and emptying my database?

Answer 1

There is also something called "logicalFilePath" on liquibase which is described as:

Use to override the file name and path when creating the unique identifier of change sets. Required when moving or renaming change logs.

Maybe that helps. If I remember it correctly this option allows to not include the complete file path to the identifier.

(This does not work for formatted sql files though. See issue: CORE-915 in Liquibase Jira.

Answer 2

I like to point to old conversation between upstream and users:

http://forum.liquibase.org/topic/why-does-the-change-log-contain-the-file-name

I think that upstream arguments are weak and they make dumb decition to relay on full file path. Initial idea that you fix file paths via CLASSPATH but your request shown that this is wrong also.

Upstream developers suggest to perform direct update on DATABASECHANGELOG.FILENAME column to fix broken entries with full paths.

If you set hashes DATABASECHANGELOG.MD5SUM ot null that trigger hashes recalculation on next LiquiBase run. You should do this because hash algorithm uses all compound ID parts when calculated.

logicalFilePath can be applied to top level XML tag:

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd"
                   logicalFilePath="legacy.xml">
    ...
</databaseChangeLog>

or may be overridden by each changeset:

<changeSet author="admin" id="fix-25" logicalFilePath="fix.xml">
   ...
</changeSet>

SQL syntax also have logicalFilePath attribute for top level file mark (implemented in v3.3):

--liquibase formatted sql  logicalFilePath:legacy.sql

and may be overridden in each changeset mark::

--changeset db-maint:tune-indexed  logicalFilePath:other.sql

To verify that settings have effect I use:

mvn liquibase:changelogSyncSQL

and review migration.sql. Corresponding pom.xml part:

<plugin>
    <groupId>org.liquibase</groupId>
    <artifactId>liquibase-maven-plugin</artifactId>
    <version>${liquibase.version}</version>
    <configuration>
        <changeLogFile>${basedir}/src/main/resources/sql/master.xml</changeLogFile>
        <propertyFile>${liquibase.profile}</propertyFile>
        <promptOnNonLocalDatabase>false</promptOnNonLocalDatabase>

        <migrationSqlOutputFile>migration.sql</migrationSqlOutputFile>
    </configuration>
</plugin>

Answer 3

Liquibase has a "changelogSync" command, which can be used to mark changesets as executed in the database.