Azure DevOps Server 2019 Retention Policy no longer working

2
votes

Last week we have migrated from TFVC to Git on our ADS 2019.1 server.

On our validation pipeline we have an aggressive retention policy. It is set to Keep for 2 days, 10 good builds with the branch filter * and all checkboxes cleared ADS writes it as: +refs/heads/*

What we see is that since we migrated, none of the Git builds have been deleted. From the last 8 days we now have hundreds of builds with their multi-GB drop.

Is * the right branch filter to use ? This was the default setting entered by ADS, we do indeed want all builds for all branches to be deleted by this policy.

I also tried **, but that made no change, in the build results page i see all build are build off a temporary branch with a 4 digit number, which is probably the pull request number.

How can we ensure the builds are cleaned up as per the retention policies ?

The Build definition is not linked to a Release pipeline. The builds are triggered from the Pull Requests created by developers. The Pull requests are marked as Completed.

Another build definition, one with a CI trigger that runs after every commit, does still seem to delete builds based on the retention policy defined.

Picture of our TFS Retention policy settings. Master builds are cleaned up, pull request builds are not Retention policy settings Pull request build

List of the oldest builds in TFS after manual cleanup last week, these builds are now 4-5 days old, over the 2 day limit of the retention policy. Also you can see there is no retain lock behind them. Oldest builds under build history

Overview of the deleted builds. You can see 'master' builds in there that have been deleted by the retention policy. Latest deleted builds

Shows the master builds are deleted after the 2 day retention policy setting Deleted build by retention

3
tfsazure-devopsazure-pipelines
Do you use release pipelines? What is the retention on your release pipelines. If you have selected to keep artifacts on the release pipelines, it will use the release retention and not the build retention. – Matt
Hi @Matt, no we do not use Release pipelines for this definition, i have added more details to the original post – Nico
Why not remove all branch filters if you want this to apply to all. – jessehouwing
That is because TFS enforces you to have one "You must add at least one branch filter." – Nico

3 Answers

2
votes

You may also try:

  1. Tab "Retention"
  2. Select a policy (in your case: "Keep for 2 days, […]")
  3. Under "Branch filters", add another one
  4. Type: "Include", Branch specification: "refs/pull/*"

Explaination: If you open a terminal in the build agent's source folder (named s) and run git branch --all, you will see a lot of branches like refs/pull/1234 for each pull request (e.g. 1234) there. Also, if you compare your branch specification * to the path shown on the left (+refs/heads/*) you see that it does in fact not match all branches…

0
votes

Azure DevOps Server 2019 Retention Policy no longer working

Agree with Matt. If the artifacts of the release pipeline is Build, the build pipeline will use the release retention instead of the build retention:

enter image description here

So, we need to check the release retention, Project Settings->Release retention:

You could check this info from the document Build and release retention policies-Q&A.

Besides, if the build to be retained indefinitely, the retention policy will no longer apply.

Hope this helps.

0
votes

The retention policy UI is extremely misleading, because * is not really * - it's refs/heads/*. When the UI (or perhaps the API) saves the 'branch specification' it implicitly adds +refs/heads/ to whatever you enter. main becomes +refs/heads/main, mybranch/* becomes +refs/heads/mybranch/*.

You can see this by looking at the JSON in the RetentionPolicy column in [Build].[tbl_Definition] (assuming default collection DB name):

SELECT [DefinitionName],[RetentionPolicy]
FROM [Tfs_DefaultCollection].[Build].[tbl_Definition]

The default build retention policy, which will never match pull request branches, is stored as:

[
  {
    "branches": [
      "+refs/heads/*"
    ],
    "artifacts": [],
    "artifactTypesToDelete": [
      "FilePath",
      "SymbolStore"
    ],
    "daysToKeep": 10,
    "minimumToKeep": 2,
    "deleteBuildRecord": true,
    "deleteTestResults": true
  }
]

The TFS Job Agent's build retention policy job calls the stored procedure [Build].[prc_GetBuildsForRetention] once per build definition to get a list of potential builds to clean up. Builds that are retained by a release are already excluded from the query results.

You can try it on your own with a query like this (update the definitionId and maxFinishTime, and possibly the dataspaceId and partitionId):

EXEC [Tfs_DefaultCollection].[Build].[prc_GetBuildsForRetention]
    @partitionId = 1,
    @dataspaceId = 22,
    @definitionId = 164,
    @minFinishTime = '0001-01-01',
    @maxFinishTime = '2021-08-03',
    @maxBuilds = 1000

You'll see results similar to this:

prc_GetBuildsForRetention results

To include pull requests in your retention policy, add a new entry and type refs/pull/* into the branch specification box. That will persist into the JSON as +refs/pull/*. You'll probably want to set 'days to keep' low (I use 1), and 'minimum to keep' low (I use 0). Our PR builds are set to expire in a few hours, so there's little benefit in keeping them longer.

For future reference, the build retention policy job code is located in <installFolder>\Application Tier\TFSJobAgent\Plugins\Microsoft.TeamFoundation.Build2.Server.Extensions.dll in the Microsoft.TeamFoundation.Build2.Server.Extensions.BuildRetentionPolicyJob class.