Keystore Migration Troubleshooting

Symptoms

  • Collector or Server is not starting.
  • Jetty for DTANG, web services, or WebUI is not starting.

Also, one of the Logfiles in:

  • DT_HOME/log/collector/NAME/Collector.X.Y.log
  • DT_HOME/log/server/Server.X.Y.log
  • DT_HOME/log/server/FrontendServer.X.Y.log
  • DT_CLIENT_HOME/log/client/Client.X.Y.log
    Has one or several of the following strings:
    • The keystore file 'XXXX' is invalid
    • BrokenKeystoreFileException
    • javax.crypto.BadPaddingException: pad block corrupted
    • The subsequent java.lang.NullPointerException and java.security.UnrecoverableKeyException may differ a little, depending on the component (server, frontend server, collector, client):
      • [AbstractLifeCycle] FAILED SslContextFactory@47252b0d(/home/dynatrace/dynatrace-7.0.0/server/conf/jetty.jks,/home/dynatrace/dynatrace-7.0.0/server/conf/jetty.jks): java.lang.NullPointerException: org.eclipse.jetty.util.log.JavaUtilLog warn:71 java.lang.NullPointerException
        at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:911)
      • [AbstractLifeCycle] FAILED SslContextFactory@1190d87(/home/dynatrace/dynatrace-7.0.0/server/conf/jetty_fe.jks,/home/dynatrace/dynatrace-7.0.0/server/conf/jetty_fe.jks): java.security.UnrecoverableKeyException: Password must not be null: org.eclipse.jetty.util.log.JavaUtilLog warn:71 java.security.UnrecoverableKeyException: Password must not be null

Solution 1

  1. Download new version of the migration tool.
  2. Recreate migration archive and re-try.

Solution 2

  1. Shutdown all components using the affected installation directory.
  2. Backup the full <DT_HOME>/conf folder.
  3. Execute steps that apply.

Troubleshooting dt_pwdstore

This troubleshooting applies to dt_pwdstore files only.

In this case 'XXXX' in the exception is one of the following:

  • dt_pwdstore.bks/salt/key
  • java.lang.NullPointerException at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:911)
  • java.security.UnrecoverableKeyException: Password must not be null
  1. Delete all dt_pwdstore.* and .dt_pwdstore.*.bkp files (the default dt_pwdstore is extracted at next startup) if you have a normal deployment.
  2. Extract the dt_pwdstore.zip if you have a customized dt_pwdstore.
  3. If still failing, delete all keystores, including *.bks, *.salt, *.key, and *.bkp keystore files in the folder.

Data lost:

  • Nothing.
  • All re-initialized stock passwords. Restore these by re-extracting dt_pwdstore.zip, or copy all dt_pwdstore.* files from a working component.
  • Step 3 only: See sections of the specific component there data is lost.

Troubleshooting Collector

This troubleshooting applies to Collector only.

In this case 'XXXX' in the exception = dt_co_keystore.bks/salt.

The migration tool may copy potentially corrupt files from older deployments and leave needed ones behind.

  1. If .dt_co_keystore.*.bkp files are present, restore them by removing '.' from the prefix and '.bkp' from the suffix.
  2. Start the component.
  3. If still failing, delete *.bks, *.salt, *.key, *.bkp.

Data lost:

  • Collector proxy password.
  • Private key and certificate of the collector.
  • Accepted certificates from the AppMon server.

Troubleshooting Client

This troubleshooting applies to the AppMon Client only.

In this case 'XXXX' in the exception = dt_cl_keystore.bks/salt

Client keystores may be corrupted.

  1. Delete dt_keystore.*, .dt_keystore.*.bkp, dt_cl_keystore.*.
  2. In case .dt_cl_keystore.*.bkp files are present, restore them by removing . from the prefix and .bkp from the suffix.
  3. Start the component. If still failing, delete *.bks, *.salt, *.key, *.bkp.

Data lost:

  • Collector proxy password.
  • Private key and certificate of the Collector.
  • Accepted certificates from the AppMon server.

Troubleshooting Frontend Server

This troubleshooting applies to Frontend Server only.

In this case 'XXXX' in the exception = dt_fe_keystore.bks/salt.

The migration tool may copy potentially corrupt files from older deployments and leave needed ones behind.

  1. Delete dt_fe_keystore.* if still present.
  2. If .dt_fe_keystore.*.bkp files are present, restore them by removing '.' from the prefix and '.bkp' from the suffix. Otherwise, simply start the component.
  3. If it's still failing and .dt_keystore.*.bkp files are present, delete dt_keystore.bks and dt_keystore.salt, restore the .bkp files and restart.
  4. If still failing, delete *.bks, *.salt, *.key, *.bkp.

Data lost:

  • Private key and certificate of the frontend server.
  • Accepted certificates from the backend server.

Troubleshooting Backend Server

This troubleshooting applies to Backend Server only.

In this case 'XXXX' in the exception = dt_keystore.bks/salt.

The migration tool may copy potentially corrupt files from older deployments and leave needed ones behind.

  1. Delete dt_keystore.* if still present.
  2. If .dt_keystore.*.bkp files are present, restore them by removing '.' from the prefix and '.bkp' from the suffix. Otherwise, start the component.
  3. If still failing, delete *.bks, *.salt, *.key, *.bkp.

Data lost:

  • Passwords like Gomez integration, PWH integration, SMTP integration, BT Export, or Analysis Stream.
  • Private key and certificate of the backend server.
  • Accepted certificates from the memory analysis server.

Troubleshooting Memory Analysis Server

This troubleshooting applies to Memory Analysis Server only.

In this case 'XXXX' in the exception = dt_al_keystore.bks/salt.

The migration tool may copy potentially corrupt files from older deployments and leave needed ones behind.

  1. Delete dt_al_keystore.* if still present.
  2. If .dt_al_keystore.*.bkp files are present, restore them by removing '.' from the prefix and '.bkp' from the suffix, Otherwise simply start the component.
  3. If it's still failing and .dt_keystore.*.bkp files are present, delete dt_keystore.bks and dt_keystore.salt, restore the .bkp files restart.
  4. If still failing, delete *.bks, *.salt, *.key, *.bkp.

Data lost:

  • Private key and certificate of the memory analysis server.