Mirror Maven Repository using OSS Software (tried using Artifactory OSS)

3
votes

Quick overview of the status quo:

  • I have a corporate maven repository that is hosted at a different location
  • The internet connection to said repository is pretty slow
  • I would like to host a cache/mirror at my site so my and my team can download dependencies faster
  • We have a single dependency that stands out because it is 8GB in size which means it takes around 40min just to download

So I tried to use Artifactory OSS, but:

  • The official 'repository replication' feature is locked behind a >=3k$/year paywall. It's basically the same as a 'remote repository' that caches proactively in one way or another. Pretty much the same situation with Sonatype Nexus OSS.
  • I would generally be okay with using a plain 'remote repository' in Artifactory, but it behaves weirdly.
  • The intended functionality is that a client requests a dependency from the on-site Artifactory remote repository pointing to the off-site actual server. If it's not cached, the on-site server downloads it from the off-site server. Then it's served to the client by the on-site server
  • When a large dependency is requested, the on-site server starts downloading and after ~30s the client gets a socket timeout and stops the entire gradle build
  • There is an option called 'store locally' that when deactivated forwards the request to the off-site server and doesn't cache at all
  • What I'd want is the on-site server to start caching when a dependency is requested, but responding that the dependency is not present on this server (causing the client to look at the off-site server) or forward the request to the off-site server like it does when 'store localy' is disabled

After all I'm not sure where the problem is. Do I have to configure Artifactory or Gradle differently? Is there something else I could use to create a mirror? A totally acceptable solution would be to run a copy-script every night, but I'd prefer solving it with Artifactory OSS.

1
javamavengradleserverartifactory
Usually you host a local cache which mirrors the content of Central. This content is downloaded on demand....Furthermore you have a dependency in your own location which is 8 GiB size? Really? Which you are using as a dependency? sounds weird...Usually you are running a Nexus as a mirror/cache not as a replication of Central...which is without cost ?...? - khmarbaise

1 Answers

0
votes

I am relatively sure that what you want is not possible with either Artifactory or Nexus.

Your requirements are rather unusual. I would consider a dependency with 20MB "large", and I have never seen one over 100MB. Your 8GB thing is not a jar, right? Nexus/Artifactory is not made for this kind of stuff.

First of all, I would question the 8GB thing. What is it? Why is it so large?

If you cannot change that, I would run a mvn dependency:get or something similar on the build server in regular intervals. This way, your on-site Artifactory/Nexus would (usually) have the relevant version present and your build will not time out.