Upgrade of datastax-agent fails from opscenter

Question

Running Datastax Enterprise 4.5.1 with OpsCenter 5.0.0 and attempting to upgrade OpsCenter and datastax-agents to OpsCenter version 5.1.0

This is a test cluster of 2 nodes in AWS.

I downloaded OpsCenter 5.1.0 from the Datastax website and successfully upgraded the OpsCenter server in my test cluster. As expected, after connecting to the new OpsCenter webpage, a message was displayed saying that the agents couldn't connect.

I attempted to upgrade the agents from the OpsCenter web UI. I added the credentials for AWS, username 'ubuntu' and public key in pem format and ran the upgrade. It failed.

In /var/log/opscenterd/opscenterd.log, I see the following:

2015-03-05 17:47:33+0000 []  WARN: Marking request e3f82540-3bd8-4d27-81dd-e40fd476dd0a as failed: Failure installing agent on 10.0.11.138.
         Standard output:
                bin/find-java
                bin/install_agent.sh
                bin/sudowrap
                ./datastax-agent.deb
                ./datastax-agent.rpm
                DataStax AMI wasn't used
                * Stopping DataStax Agent datastax-agent
                ...done.
                Reading package lists...
                Building dependency tree...
                Reading state information...
                E: Unable to locate package opscenter-agent
                Starting agent installation process for version 5.1.0
                Reading package lists...
                Building dependency tree...
                Reading state information...
                sysstat is already the newest version.
                0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
                Selecting previously unselected package datastax-agent.
                (Reading database ... 82449 files and directories currently installed.)
                Preparing to unpack .../datastax-agent.deb ...
                Unpacking datastax-agent (5.1.0) ...
                Setting up datastax-agent (5.1.0) ...

                Configuration file '/etc/datastax-agent/datastax-agent-env.sh'
                ==> File on system created by you or by a script.
                ==> File also in package provided by package maintainer.
                What would you like to do about it ?  Your options are:
                Y or I  : install the package maintainer's version
                N or O  : keep your currently-installed version
                D     : show the differences between the versions
                Z     : start a shell to examine the situation
                The default action is to keep your current version.

         Exit code: 130

When I went to the nodes, the dpkg process was still running and I had to kill it manually. I ran the upgrade procedure from the webUI again; this time it says it was successful, but the agents are still not connecting.

Any idea what happened, or how to resolve it?

Is the agent running on your nodes? ps -ef | grep datastax-agent — phact
why are there 2 jars? opscent+ 20387 1 41 20:52 pts/0 00:00:07 /usr/lib/jvm/java-7-oracle/jre/bin/java -Xmx40M -Xms40M -Dopscenter.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStorePassword=opscenter -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid -Dlog4j.configuration=./conf/log4j.properties -Djava.security.auth.login.config=./conf/kerberos.config -jar datastax-agent-5.0.0-standalone.jar datastax-agent-5.1.0-standalone.jar /var/lib/datastax-agent/conf/address.yaml — LHWizard
This looks normal. Given the warning above, is your configuration file malformed? '/etc/datastax-agent/datastax-agent-env.sh' — phact
on 1st node: JVM_OPTS="$JVM_OPTS -Xmx40M" JVM_OPTS="$JVM_OPTS -Xms40M" on 2nd node, it's different: JVM_OPTS="$JVM_OPTS -Xmx128M -Djclouds.mpu.parts.magnitude=100000 -Djclouds.mpu.parts.size=16777216" — LHWizard
As I see it, apt (or dpkg) is asking a question about the new files being installed, and freezing until it gets a response. This causes the install to timeout and fail. — LHWizard

mbulman mbulman · Accepted Answer · 2015-03-17T14:23:17

This is a known issue when configuration files that have changed in the packages are modified by the user. The two possible workarounds for this are to manually install the agent package on each node, so that you can choose what action the package manager should take; or to revert the changes in datastax-agent-env.sh, install the agents through OpsCenter, and then re-apply your changes (this is only possible if you know exactly what changes were made).

For reference, the internal ticket for fixing this in OpsCenter is OPSC-4495, which should be fixed in an upcoming patch release.

Upgrade of datastax-agent fails from opscenter

2 Answers