Upgrade custom Dynatrace extensions from Python 3.6 to Python 3.8
Starting with Dynatrace OneAgent and ActiveGate version 1.231, we are upgrading the Extension Framework (also referred to as the plugins framework) from Python 3.6 to Python 3.8.
- Consequences: Some Dynatrace extensions running in your environment may stop working and require redeployment of a new version prepared for Python 3.8.
- Symptoms:
No data is provided for affected metrics on dashboards, alerts, and custom device pages populated by the affected extension metrics.
- Extension logs display errors. Most often this will manifest itself as Python
ModuleNotFoundErrorin the PluginAgent log. Sometimes the Python virtual machine crashes.
- Impact: This issue affects only those extensions that use native libraries called from Python code distributed with the extension.
Recommended solution
In case of any problems with custom extensions after upgrade to OneAgent version 1.231+, we recommend that you rebuild the extension package using Python 3.8 together with the newest modules and redeploy it to all hosts.
For details on building, uploading, and deploying extensions, see Extensions 1.0.
Note that Python 3.8 must be used for building the extension, meaning that plugin_sdk has to be installed in a Python 3.8 environment. We recommend using Python Virtual Environment prepared by Python 3.8.
Extensions running on Windows machines have to be rebuilt on Windows; Linux extensions have to be rebuilt on Linux.
Only 64-bit platforms are currently supported.
Update the extension version and install_requires properties in the plugin.json file.
The dependency modules defined in the install_requires property should be the latest available, or versions that are supported by Python 3.8.
Migration example
The following is an excerpt from the plugin.json file for a Windows-specific extension.
{"name": "custom.python.mssql","version": "1.0","type": "python","requiredAgentVersion": "0.000.0","entity": "PROCESS_GROUP_INSTANCE","technologies": ["MSSQL"],"source": {"package": "custom_mssql_plugin","className": "CustomMsSqlPlugin","install_requires": ["pymssql==2.1.4"],"activation": "Singleton"},
-
This is version 1.0; it doesn't require any specific OneAgent version.
-
The extension requires pymssql module version 2.1.4.
-
The extension package contains platform-specific binary modules:
_mssql.cp36-win_amd64.pydpymssql.cp36-win_amd64.pyd
These modules support Python versions 3.7 and earlier, not Python 3.8 (see pymssql 2.1.4).
Error message
The extension will stop working after upgrade to OneAgent version 1.231+ and the following error will appear in the PluginAgent log.
2021-10-26 11:03:32.032 UTC [00000c5c] info [native] 3164(MainThread) - [report_status] status changed for engine ('custom.python.mssql', 16080871464060409180) PluginFullStatus(pluginName=custom.python.mssql, pluginVersion=1.0, state=ERROR_UNKNOWN, description=No module named 'pymssql', monitoredEntityId=0xdf2abb657e09555c, stacktrace=Traceback (most recent call last):File "C:\ProgramData/dynatrace/oneagent/agent/runtime/engine_unpacked\ruxit\plugin_state_machine.py", line 336, in _execute_next_taskself._query_plugin()File "C:\ProgramData/dynatrace/oneagent/agent/runtime/engine_unpacked\ruxit\plugin_state_machine.py", line 663, in _query_pluginself._plugin_run_data = self._create_plugin_run_data()File "C:\ProgramData/dynatrace/oneagent/agent/runtime/engine_unpacked\ruxit\plugin_state_machine.py", line 636, in _create_plugin_run_dataplugin_module = importlib.import_module(self.metadata["source"]["package"])File "C:\Program Files/dynatrace/oneagent/agent/plugin/python3.8\importlib\__init__.py", line 127, in import_modulereturn _bootstrap._gcd_import(name[level:], package, level)File "<frozen importlib._bootstrap>", line 1014, in _gcd_importFile "<frozen importlib._bootstrap>", line 991, in _find_and_loadFile "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlockedFile "<frozen importlib._bootstrap>", line 671, in _load_unlockedFile "<frozen importlib._bootstrap_external>", line 843, in exec_moduleFile "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removedFile "C:\Program Files/dynatrace/oneagent/plugin_deployment/custom.python.mssql\custom_mssql_plugin.py", line 14, in <module>import pymssqlModuleNotFoundError: No module named 'pymssql')
The extension must be updated and re-built using Python 3.8 and the newest pymssql module, which is currently ver. 2.2.2. The new version of the extension package has to be deployed to all hosts where it runs.
Resolution
Update the extension definition by modifying plugin.json as follows:
{"name": "custom.python.mssql","version": "1.1","type": "python","requiredAgentVersion": "1.231.0","entity": "PROCESS_GROUP_INSTANCE","technologies": ["MSSQL"],"source": {"package": "custom_mssql_plugin","className": "CustomMsSqlPlugin","install_requires": ["pymssql==2.2.2"],"activation": "Singleton"},
where properties version, requiredAgentVersion, and install_requires are updated.
Now the extension is ready to be rebuilt. See plugin_sdk build_plugin in OneAgent extensions tutorial.
Verify that the new extension package contains updated modules:
_mssql.cp38-win_amd64.pyd_pymssql.cp38-win_amd64.pyd
Deploy the package to the OneAgent host, restart OneAgent service, and verify in the PluginAgent log that the extension starts successfully.