How can I use Git token to access private repository with Travis CI / Cloudfoundry?

Question

Config

---

CF CLI version cf version

cf version 6.37.0+a40009753.2018-05-25

Buildpack version

https://github.com/cloudfoundry/nodejs-buildpack

Manifest

applications:
- path: .
  memory: 2048M
  instances: 1
  buildpack: nodejs_buildpack
  name: kpb-singlenode-api-tmp
  command: node server.js
  disk_quota: 2048M

deploy.sh

#!/bin/bash
./Bluemix_CLI/bin/ibmcloud config --check-version false
./Bluemix_CLI/bin/ibmcloud api $API_ENDPOINT
./Bluemix_CLI/bin/ibmcloud login --apikey $API_KEY
./Bluemix_CLI/bin/ibmcloud target -o $IBMCLOUD_ORGANIZATION -s $IBMCLOUD_SPACE
./Bluemix_CLI/bin/ibmcloud app push kpb-node-api

.travis.yml

language: node_js
node_js:
  - '8'
script: echo "skipping tests"
before_deploy:
  - curl -L https://clis.ng.bluemix.net/download/bluemix-cli/latest/linux64 | tar -zx
  - chmod -R u+x ./Bluemix_CLI/bin
  - chmod +x ./deploy.sh
deploy:
  provider: script
  script: ./deploy.sh
  on:
    repo: myrepo/kpb-node-api
    branch: master
  skip_cleanup: true

Issue

---

I'm just trying to push my application on IBM Cloud (cloudfoundry) but I am using private repositories on github Enterprise hence cf (cloudfoundry) building agent fails npm install as it tries login/password connection (which is denied) while it should use a Git token...

The build is automated with Travis CI.

Expected behavior

Cloudfoundry (or Travis?) agent should use git token while running npm install

Actual behavior

It sticks to login/password credentials so github throws you should use git token or ssh key instead

---

As far as I know the problem is that we are using a private repository, declared like this: git+https://github.com/someone/awesome-private-pkg.git (we can't use npm publish etc...) The error will be thrown while cloudfoundry tries to npm install the private repository with login/password credentials

This is my error logs:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.ibm.com/myrepo/kpb-api-pkg
npm ERR! 
npm ERR! remote: Password authentication is not available for Git operations.
npm ERR! remote: You must use a personal access token or SSH key.
npm ERR! remote: See https://github.ibm.com/settings/tokens or https://github.ibm.com/settings/ssh
npm ERR! fatal: unable to access 'https://github.ibm.com/myrepo/kpb-api-pkg/': The requested URL returned error: 403
npm ERR! 
npm ERR! exited with error code: 128
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2018-06-26T10_31_07_934Z-debug.log

I'm digging on .bashrc to maybe set the vars via git config --global git.token

Thank you for helping, have a nice day!

Answer 1

So you just have to add the following:

before_install:
  - echo -e "machine github.ibm.com\n  login $GIT_TOKEN" > ~/.netrc

to your .travis.yml

---

Solution kind of came by itself, from Travis docs

This table is pretty explicit in terms of access, as I was digging into fetching all my private modules with SSH Deploy Key method (git+ssh://[email protected]/org/app), it cames pretty difficult to get all the repos with a single SSH key...

Hence they suggest the User Key method which is the best but that I can't apply because of GitHub Enterprise which binds 1 company mail address to 1 GHubE account (SAML stuff)

Like I was saying in my post I wasn't able to provide the right creds the right way I didn't figure out that a .netrc file exists according to Password and API token methods

Apparently it sets travis agent to use the desired login type (except for ssh)!

---

Big thanks to @DanielMikusa for his help!

Answer 2

There's a lot of info here and I'm not sure it's all relevant. It seems that the crux of your problem is that when your application is staged (i.e. when the buildpack runs), the buildpack runs npm install which in turn tries to grab a dependency from a private github repo.

Assuming I've got that correct...

1. One possible solution would be to vendor your dependencies. When you "vendor" dependencies, you're basically running npm install locally or in some environment you control, then you're sending the dependencies up with your application to Cloud Foundry. It makes for a larger application, but when your application stages and the platform runs npm install all the dependencies exist already so npm install becomes basically a no-op.

   The main trick is with compiled dependencies. To make this work, you need to vendor your dependencies on a system that matches the rootfs on Cloud Foundry. Currently the cflinuxfs2 stack matches Ubuntu Trusty 14.04.

   You can read more about vendoring dependencies in the docs here.

   https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring

2. Another solution would be to use cf local. This is a cf cli plugin that uses Docker to run buildpacks on your local system. On the local system, you'd presumably be able to access your private repository (it's basically running from your laptop/PC). The buildpack would then run and create a completely self-contained droplet. You can then use cf local to export that droplet & send it to any other Cloud Foundry platform.

   The beauty is that the droplet is everything that's needed for the platform to run your app, so it doesn't even need to stage your application which bypasses the who problem of accessing your private code from the staging environment on the public Cloud Foundry.

3. The other thing you might be able to do, and I'm not 100% sure this will work, is to include a .npmrc file. The buildpack should configure npm to look at this file when it runs npm install. The idea here would be to include some configuration option to npm that would allow it to connect to your repo. I'm not an expert with npm though, so I don't know if such an option exists. Just thought I'd mention this in case it might help.