Finally I have automated this migrations process as much as possible in the following way:
1) Download the project from StarTeam
2) I have Developed a JAVA utility which will scan the project workspace and dump all the jar details into an excel sheet. For each jar it will calculate the checksum (SHA-1) using the below code
public static String calculateChecksum(File javaJarFile) {
String filepath = javaJarFile.getAbsolutePath();
StringBuilder sb = new StringBuilder();
FileInputStream fis = null;
try {
MessageDigest md = MessageDigest.getInstance("SHA-1"); //40 character checksum
fis = new FileInputStream(filepath);
byte[] dataBytes = new byte[1024];
int nread = 0;
while ((nread = fis.read(dataBytes)) != -1)
md.update(dataBytes, 0, nread);
byte[] mdbytes = md.digest();
for (int i = 0; i < mdbytes.length; i++)
sb.append(Integer.toString((mdbytes[i] & 0xff) + 0x100, 16)
.substring(1));
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fis != null)
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Then it will query the Artifactory using the checksum value for the matching jar details like groupId, artifactId, version etc...If the jar details not found in Artifactory, then it will query MAVEN central repository for the matching jar details. Finally all the existing project jar details and the corresponding MAVEN compatible jar details will be dumped in the excel sheet.
Sample code for querying the MAVEN central repository
CloseableHttpClient httpClient = HttpClients.custom().build();
HttpPost getRequest = new HttpPost("http://search.maven.org/solrsearch/select?q=1:<JAR CHECKSUM>&rows=1&wt=json");
getRequest.addHeader("accept", "application/json");
HttpResponse response = httpClient.execute(getRequest);
if (response.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(response.getEntity().getContent())));
String output;
StringBuffer outputBuffer = new StringBuffer("");
while ((output = br.readLine()) != null) {
outputBuffer.append(output);
}
JSONObject jsonObj = new JSONObject(outputBuffer.toString());
LOGGER.info("MAVEN Compatible Dependency Found: " + jsonObj.getJSONObject ("response").getInt("numFound"));
if (jsonObj.getJSONObject ("response").getInt("numFound") > 1) {
JSONArray jsonArray = jsonObj.getJSONObject ("response").getJSONArray("docs");
JSONObject object = (JSONObject) jsonArray.get(0);
LOGGER.info(object.getString("g"));
LOGGER.info(object.getString("a"));
LOGGER.info(object.getString("v"));
}
Sample code for querying the Artifactory
String checkSumUri = "https://anupg.jfrog.io/anupg/api/search/checksum?sha1=" + checkSum;
HttpResponse rtFactResponse = callService (checkSumUri, true);
BufferedReader br = new BufferedReader(new InputStreamReader((rtFactResponse.getEntity().getContent())));
String output;
StringBuffer outputBuffer = new StringBuffer("");
while ((output = br.readLine()) != null) {
outputBuffer.append(output);
}
String uri = null;
if (!outputBuffer.toString().trim().equals("")) {
JSONObject jsonObj = new JSONObject(outputBuffer.toString());
JSONArray jsonArray = jsonObj.getJSONArray("results");
for (int i=0; i < jsonArray.length(); i++) {
JSONObject jsonUriObject = (JSONObject) jsonArray.get(i);
System.out.println("URI---------->" + jsonUriObject.getString("uri").replace(".jar", ".pom"));
uri = jsonUriObject.getString("uri").replace(".jar", ".pom");
}
}
if (uri != null) {
String downloadURI = null;
HttpResponse uriResponse = callService (uri);
BufferedReader uriBR = new BufferedReader(new InputStreamReader(
(uriResponse.getEntity().getContent())));
String uriOutput;
StringBuffer uriOutputBuffer = new StringBuffer("");
while ((uriOutput = uriBR.readLine()) != null) {
uriOutputBuffer.append(uriOutput);
}
if (!uriOutputBuffer.toString().trim().equals("")) {
JSONObject jsonUriObject = new JSONObject(uriOutputBuffer.toString());
System.out.println("Download URI---------->" + jsonUriObject.getString("downloadUri"));
downloadURI = jsonUriObject.getString("downloadUri");
}
HttpResponse downloadUriResponse = callService (downloadURI, true);
if (downloadUriResponse.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ downloadUriResponse.getStatusLine().getStatusCode());
}
InputStream is = downloadUriResponse.getEntity().getContent();
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
Document doc = null;
try {
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
doc = dBuilder.parse(is);
doc.getDocumentElement().normalize();
} catch (ParserConfigurationException e) {
e.printStackTrace();
} catch (SAXException e) {
e.printStackTrace();
} finally {
if (is != null) {
is.close();
}
}
System.out.println("root of xml file: " + doc.getDocumentElement().getNodeName());
Element element = doc.getDocumentElement();
if (getNodeValue("groupId", element) != null &&
getNodeValue("artifactId", element) != null &&
getNodeValue("version", element) != null) {
metadata.setGroupId(getNodeValue("groupId", element));
metadata.setArtifactId(getNodeValue("artifactId", element));
metadata.setVersion(getNodeValue("version", element));
metadata.setRTfactoryFound(true);
}
}
There could be a chance that project may contain custom jars which may not be available either in Artifactory or MAVEN central. In such cases the generated excel sheet will be shared with AD team for corresponding jar details. The jar will be installed in Artifactory and the AD team will update the excel sheet with the groupId, arfactId, version.
Once the excel sheet having all the jar details the JAVA utility will generate the pom.xml by reading the corresponding MAVEN details.
The "copy-rename-maven-plugin" plugin is used here to copy all the maven downloaded jars from "target/dependency" folder to respective project folder.
This pom will be configured as the first build steps in Bamboo server followed by build.xml which will build the project. Note I am using hybrid approach in build process.
Below is the code snippets for the same
Writer w = null;
MavenXpp3Writer mavenXpp3Writer = new MavenXpp3Writer();
try {
List<JarDetails> uniquejarDetails = removeDuplicateJars (jarDetails);
w = WriterFactory.newWriter (new File(location + "pom.xml"), "UTF-8");
Model model = new Model();
model.setGroupId("com.projectname");
model.setArtifactId("project-analyzer");
model.setVersion("1.0");
model.setModelVersion("4.0.0");
List<Dependency> dependencies = new ArrayList<Dependency>();
Plugin copyDependency = new Plugin();
copyDependency.setGroupId("org.apache.maven.plugins");
copyDependency.setArtifactId("maven-dependency-plugin");
copyDependency.setVersion("2.10");
PluginExecution copyDependencyPluginExecution = new PluginExecution();
copyDependencyPluginExecution.setId("copy-dependencies");
copyDependencyPluginExecution.setPhase("generate-sources");
List<String> copyDependencyGoalsList = new ArrayList<String>();
copyDependencyGoalsList.add("copy-dependencies");
copyDependencyPluginExecution.setGoals(copyDependencyGoalsList);
Plugin plugin = new Plugin();
plugin.setGroupId("com.coderplus.maven.plugins");
plugin.setArtifactId("copy-rename-maven-plugin");
plugin.setVersion("1.0.1");
PluginExecution pluginExecution = new PluginExecution();
pluginExecution.setId("copy-jars");
pluginExecution.setPhase("generate-sources");
List<String> goalsList = new ArrayList<String>();
goalsList.add("copy");
pluginExecution.setGoals(goalsList);
String domString = "<configuration><fileSets>";
for (int jarCount = 0; jarCount < uniquejarDetails.size(); jarCount++) {
if (uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId() != null) {
Dependency dependency = new Dependency();
dependency.setGroupId(uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId());
dependency.setArtifactId(uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId());
dependency.setVersion(uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion());
dependencies.add(dependency);
//Add copy-rename-maven-plugin configurations
String mavenJarName = uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId() + "-"
+ uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion() + ".jar";
String mavenJar = "target/dependency/" + mavenJarName;
domString += "<fileSet><sourceFile>" + mavenJar + "</sourceFile>";
domString += "<destinationFile>" + new File(uniquejarDetails.get(jarCount).getJarRelativePath() +
"/" + mavenJarName) + "</destinationFile></fileSet>";
}
}
domString += "</fileSets></configuration>";
InputSource is = new InputSource();
is.setCharacterStream(new StringReader(domString));
Xpp3Dom dom = Xpp3DomBuilder.build(new StringReader(domString));
pluginExecution.setConfiguration(dom);
List<PluginExecution> pluginExecuionList = new ArrayList<PluginExecution>();
pluginExecuionList.add(pluginExecution);
List<PluginExecution> copyDependencyPluginExecuionList = new ArrayList<PluginExecution>();
copyDependencyPluginExecuionList.add(copyDependencyPluginExecution);
plugin.setExecutions (pluginExecuionList);
copyDependency.setExecutions (copyDependencyPluginExecuionList);
List<Plugin> pluginList = new ArrayList<Plugin> ();
pluginList.add(copyDependency);
pluginList.add(plugin);
Build build = new Build();
build.setPlugins(pluginList);
model.setDependencies(dependencies);
model.setBuild(build);
mavenXpp3Writer.write(w, model);
} catch (UnsupportedEncodingException e) {
LOGGER.error(e.getMessage(), e);
} catch (FileNotFoundException e) {
LOGGER.error(e.getMessage(), e);
} catch (IOException e) {
LOGGER.error(e.getMessage(), e);
} catch (FactoryConfigurationError e) {
e.printStackTrace();
} catch (XmlPullParserException e) {
e.printStackTrace();
} finally {
try {
if (w != null)
w.close();
} catch (IOException e) {
LOGGER.error (e.getMessage(), e);
}
}
Next steps is cleanup: It will delete all the jars from the project and will keep a track of the jar locations.
Created a shell script which will connect the bitbucket server and will push the code using git command. I am using "GIT Bash" to execute the script. Note, I am using Bitbucket rest API to create the project and repositories remotely before pushing the code.
REST service details for creating the project:
curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects" -d "{\"key\": \"$project_key\",\"name\": \"$project_name\", \"description\": \"$project_desc\"}" > response.json
REST service details for creating repository under the above Project
curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects/$project_key/repos" -d "{\"name\": \"$repository_name\", \"scmId\": \"git\", \"forkable\":true}" > repo-response.json
Git commands for pushing the project into Bitbucket
git init
git add .
git commit -m "Initial commit"
git remote add origin http://$bitbucket_user:${bitbucket_password}@hostname:7990/scm/${project_key}/${repository_name}.git
git push -u origin --all
- Next step is bamboo server and UrbanCode configurations which I am doing manually as bamboo server does not exposed any configurations REST API. There are REST services only for read only operations.
Using this approach we have automated 80% of the migration activity.
