Remote repository on Artifactory can not download artifact when it's not in the cache

Question

I've created a VCS repository with the name yarn-test which is pointing to github. The main goal is to use this as remote repository to releases on github.

The following URL allows us to download a release:

https://repo-url/artifactory/api/vcs/downloadRelease/yarn-test/yarnpkg/yarn/v0.23.4?ext=tar.gz

All fine. This release is downloaded and in our cache of the yarn-test registry. I can download the release from the cache using:

https://repo-url/artifactory/yarn-test/yarnpkg/yarn/releases/v0.23.4/yarn-v0.23.4.tar.gz

This seems to be good for us because we use a plugin which expects the URL of artifactory to be in a format of:

https://repo-url/artifactory/xx/xx/v0.23.4/yarn-v0.23.4.tar.gz

So when our release is in the cache of our repository it works fine. But when we upgrade the yarn release in our plugin configuration it's searching in the cache for a new version (for example v1.3.2).

It's searching for:

https://repo-url/artifactory/yarn-test/yarnpkg/yarn/releases/v1.3.2/yarn-v1.3.2.tar.gz

The URL format is good, but the v1.3.2 version is not in our cache which is normal. But here pops up our issue. We would expect it would 'translate' this to the layout of our real remote repository. But this seems not to work. We just receive a 404 error.

Why is our this not working? We can get a release from the cache but when the release does not exist our Artifactory repository is not able to download it from github because the layout is different?

Changes on our layout's do not seem to have any impact? (we really delete and recreate the remote repo with new layouts) We're using this example as inspiration:

For example, the remote repository http://download.java.net/maven/1 stores its artifacts according to the Maven 1 convention. You can configure the cache of this repository to use the Maven 2 layout, but set the Remote Layout Mapping to Maven 1. This way, the repository cache handles Maven 2 requests and artifact storage, while outgoing requests to the remote repository are translated to the Maven 1 convention.

source.

Answer 1

When accessing a VCS repository through Artifactory, the repository URL must be prefixed with api/vcs in the path.
For example, if you are using Artifactory standalone or as a local service, you would access your VCS repositories using the following URL:

http://localhost:8081/artifactory/api/vcs/<repository key>

Assuming your repository name is yarn-test, you should use the following request in order to get the v1.3.2 tag:

GET https://repo-url/artifactory/api/vcs/downloadTag/yarn-test/yarnpkg/yarn/v1.3.2

The documentation for the REST API for downloading a specific tag can be found here. For more info about VCS repositories please take a look at the documentation.

Answer 2

Repository Layouts in Artifactory are mostly used to parse/tokenize artifact paths in searches, or to derive artifact metadata from an artifact path, etc. The repository layout doesn't play any roll here.

If you want to leverage the Artifactory VCS Repo API (i.e have it send out proper requests to the Github REST API to retrieve a release archive, etc), your request has to go through the API path of that repository (meaning /api/vcs/..) and not through the direct "static" download endpoint of the repo. Artifactory won't do any automatic translation of static download URIs to an API URI.

The way I see this, you have a few options:

1. Modify your plugin to oblige to the proper structure of the URL, if possible.

2. Setup some sort of rewriting of the request URI in a reverse proxy fronting Artifactory to modify "static download" URIs to proper API URIs before the request is relayed to Artifactory.

3. Write a User Plugin to intercept an event like the "afterDownloadError" event and implement some sort of "correct URL and re-send" logic there. This particular event allows you to set both the status code and InputStream of the response payload, so you could essentially override a 404 with a proper response. Here's an example of something similar that you can base your work on:

download {
    afterDownloadError { request ->
        log.warn("Intercepting afterDownloadError event...")
        def requestRepoKey = request.getRepoPath().getRepoKey()
        def requestRepoPath = request.getRepoPath().getPath()
        if (requestRepoKey == "my-repo" && requestRepoPath.endsWith(".tar.gz")) {
            // Do something with requestRepoPath to turn it into a proper request URI to the VCS REPO
            // ...
            // Fetch
            def http = new HTTPBuilder(ARTIFACTORY_URL + requestRepoPath)
            log.warn("Directly sending a GET request to: " + ARTIFACTORY_URL + requestRepoPath)
            http.request(Method.GET, BINARY) { req ->
                response.success = { resp, binary ->
                    log.info "Got response: ${resp.statusLine}"
                    if (binary != null) {
                        // Set successful status
                        status = 200
                        // Set context-bound inputStream to return this content to the client
                        inputStream = repositories.getContent(deployRepoPath).inputStream
                        log.warn("Successfully intercepted an error on repo my-repo" +
                                " returning content from: " + ARTIFACTORY_URL + requestRepoPath)

                    } else {
                        log.warn("Received 200 response with null response content, returning 404")
                    }
                }
                response.failure = { resp ->
                    log.error "Request failed with the following status code: " + resp.statusLine.statusCode
                }
            }
        }
    }
}

Read more about User Plugins on our Wiki.

HTH,