Liquibase execution order of changeset files when using includeAll with classpath*:

Question

I am using liquibase (3.1.1) in a spring environment (3.2.x) and load the changesets via the inlcudeAll tag in a master file. There I use the "classpath*:/package/to/changesets" as path.

    <?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
         http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <includeAll path="classpath*:/package/to/changesets"/>...

I use a naming strategy like "nnn_changesetname.xml" to keep ordering. But when I look into the changeset table this ordering via the filenames are not kept. Is this only working, if the changeset files are contained in a directory and not on the classpath?

Update

Hi, I found out that the below suggested solution is not enough. I think it lies in the implementation how liquibase resolves the includAll attribute. In my case it first resolves all "folders" and then looks into each folder for changeset xmls. This will break the ordering of the xml files in all classpath*:/changes locations, because there are now several "changes" folders in different locations. What I would suspect in such a case is a merge of all contents of this "virtual" classpath folders and loading of all resources in one enumeration. Or we could allow some resouce pattern in the inlcudeAll tag like resources="classpath*:/changes/*.xml" to directly select all needed files (tried it out with the path attribute, but did not work, because it checks for a folder)?

Update

I made a hack to check if the ordering in the returned enumeration is preserved with the anwser from below. To achive this I checked for the given package name and if it matches my pattern I added an additional "*.xml" to it. With this extension I get all changeset as needed.

@Override
public Enumeration<URL> getResources(String packageName)
  throws IOException {
  if(packageName.equals("classpath*:/plugin/liquibase/changes/")) {
    packageName = packageName + "*.xml";
  }
  List<URL> resources = Collections.list(super.getResources(packageName));
  Collections.sort(resources, new Comparator<URL>() {

    @Override
    public int compare(URL url1, URL url2) {
      String path1 = FilenameUtils.getName(url1.getPath());
      String path2 = FilenameUtils.getName(url2.getPath());
      return String.CASE_INSENSITIVE_ORDER.compare(path1, path2);
    }

  });
  logger.info("Found resources: {}", resources);
  return Collections.enumeration(resources);
}};

In the log I can see now that the resources have the correct order. But when I look into the table DATABASECHANGELOCK it does not reflect the order I had in the enumeration. So it seems that this values get reodered somewhere else.

Update

Analyzed the code furhter and found out that the class liquibase.parser.core.xml.XMLChangeLogSAXHandler makes a reordering of the returned enumeration. So my changes will have no effect. I do not think that I can hack into this class as well.

Answer 1

You are right, Liquibase is relying on the underlying "list files" logic which orders files alphabetically through the file system but apparently does not through classpaths.

I created https://liquibase.jira.com/browse/CORE-1843 to track the fix.

For now, if you configure spring with a subclass of liquibase.integration.spring.SpringLiquibase that overrides getResources(String packageName) with a method that sorts the returned Enumeration that should resolve the problem for you.

Answer 2

So after some thinking and one night of sleep I came up with the following hack to guarantee order of the loaded changelog files via classpath pattern classpath*:/my/path/to/changelog/*.xml . The idea is to create the main changelog file on the fly via dom manipulation, when liquibase requests it.

It only works for the main changelog file. Following prerequisite:

• The pattern can only be used for the main changelog file
• I use an empty master changelog file as template
• All other changelog files have to use the normal allowed loading mechanism
• Works only in an Spring environment

First I had to extend/overwrite the liquibase.integration.spring.SpringLiquibase with my implementation.

public class MySpringLiquibase extends SpringLiquibase {

  private static final Logger logger = LoggerFactory.getLogger(MySpringLiquibase.class);

  private ApplicationContext context;

  private String changeLogLocationPattern;

  private List<String> changeLogLocations;

  @Autowired
  public void setContext(ApplicationContext context) {
    this.context = context;
  }

  /**
   * Location pattern to search for changelog files.
   * 
   * @param changeLogLocationPattern
   */
  public void setChangeLogLocationPattern(String changeLogLocationPattern) {
    this.changeLogLocationPattern = changeLogLocationPattern;
  }

  @Override
  public void afterPropertiesSet() throws LiquibaseException {
    try {
      changeLogLocations = new ArrayList<String>();
      // retrieve all changelog resources for the pattern
      List<Resource> changeLogResources = Arrays.asList(context.getResources(changeLogLocationPattern));
      for (Resource changeLogResource : changeLogResources) {
        // get only the classpath path of the resource
        String changeLogLocation = changeLogResource.getURL().getPath();
        changeLogLocation = "classpath:" + StringUtils.substringAfterLast(changeLogLocation, "!");
        changeLogLocations.add(changeLogLocation);
    }
    // sort all found resources by string
    Collections.sort(changeLogLocations, String.CASE_INSENSITIVE_ORDER);
  } catch (IOException e) {
    throw new LiquibaseException("Could not resolve changeLogLocationPattern", e);
  }
  super.afterPropertiesSet();
}

@Override
protected SpringResourceOpener createResourceOpener() {

  final String mainChangeLog = getChangeLog();

  return new SpringResourceOpener(getChangeLog()) {

  @Override
  public InputStream getResourceAsStream(String file)
    throws IOException {
    // check if main changelog file
    if(mainChangeLog.equals(file)) {
      // load master template and convert to dom object
      Resource masterResource = getResourceLoader().getResource(file);
      Document masterDocument = DomUtils.parse(masterResource, true);
      // add all changelog locations as include elements
      for (String changeLogLocation : changeLogLocations) {
        Element inlcudeElement = masterDocument.createElement("include");
        inlcudeElement.setAttribute("file", changeLogLocation);
        masterDocument.getDocumentElement().appendChild(inlcudeElement);
      }
      if(logger.isDebugEnabled()) {
        logger.debug("Master changeset: {}", DomUtils.toString(masterDocument));
      }
      // convert dom back to string and give it back as input resource
      return new ByteArrayInputStream(DomUtils.toBytes(masterDocument));
    } else {
      return super.getResourceAsStream(file);
    }
  }

};
}

}

This class now needs to be used in the spring xml configuration.

    <bean id="liquibase" class="liquibase.integration.spring.MySpringLiquibase"
      p:changeLog="classpath:/plugin/liquibase/master.xml"
      p:dataSource-ref="dataSource"
      p:contexts="${liquibase.contexts:prod}"
      p:ignoreClasspathPrefix="true"
      p:changeLogLocationPattern="classpath*:/plugin/liquibase/changes/*.xml"/>

With this changes I have achieved that my main changelog files are ordered by their name.

Hope that helps others too.