So since nobody else has answered this I am going to detail what I ended up doing (maybe not the best of the approaches but it works for my setup, suggestions are welcome!).
To interface with the Azure DevOps platform from a local machine you will need to configure a self-hosted agent (based on your specific OS), which will allow you to trigger builds, archive and upload the build artifacts to the Azure DevOps platform. This way you also donot have to poll for SCM changes too (which I think is not that elegant sometimes).
1. So you will need to go through the setup as outlined here for you local self-hosted agent:
NOTE: I have chosen to run the agent as service on windows for my setup
2. Next setup your Jenkins build job how you normally would, with your usual repo access setup. Things to keep in mind are following:
- Under "Build Triggers", select the Poll SCM option, but make sure that the schedule is blank, this will make sure that the trigger from your post-commit hook from the agent works. Example setup shown below:
- Under "Post-build Actions", make sure that you are archiving the artifacts as required. Example shown below:
3. Now time to setup your project's "Jenkins Service Connection", this can be accessed from the Project Settings tab on the bottom left of you project view in Azure DevOps. Note that this basically helps you self-hosted agent to locate and communicate with the Jenkins instance running locally (or an other network accessible location!). Go under Pipelines -> Service Connections and a new service connection for Jenkins. Note that the trick here is to use the URL for the connection as seen by you local self-hosted agent, which means it can be just any IP (including localhost) that the agent can access normally. Username and password are the same as the ones you setup in Jenkins. Example shown below:
NOTE: You can try to do "Verify and Save" but it will throw an error, so ignore the error or just go ahead and "Save without verification". Also you will have to do this per project, unlike the self-hosted agent setup which is per machine.
4. Now you just need to configure your build pipeline to give jobs to the right agent and pointing to the right service end-point. Now under you build pipeline settings use the agent pool that has the self-hosted agent(s) which can access your build servers. And choose the Jenkins connection that you just created in the above step. The rest of the setup is identical to how you would normally setup your project's build pipeline. An example would be as follows:
NOTE: The key here is the correct "Job name" (this should be the same as the one you have setup in you Jenkins build server instance) and the correct "Jenkins service connection".
5. The rest is straight forward in the sense that you just now need to make sure that you have a step to "Download artifacts" (NOT necessary if you donot want the artifacts on the DevOps platform) & "Publish Artifacts" (this is needed for your release pipeline to see that build artifact and to trigger it too if you want), after your jenkins queue job step. Make sure to setup the correct job directories for download from you local self-hosted agent. Example setup for both the steps:
NOTE: If you are having trouble with the paths for download and publish refer to this link for predefined variables for the self-hosted agents: https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml
6. Now in your release pipeline you should be able to add the artifact sources from you build pipeline. Example shown below:
Now you should be able to get the local artifacts in the cloud on the Azure DevOps platform, in case you cannot use the build agents provided by Microsoft for any reason!
