• Home
  • Technology support
  • Runtimes
  • Go
  • Support
  • Known limitations for Go support

Known limitations for Go support

Before you start using Go application monitoring, make sure that you are aware of the known limitations.

Support limited to official, stable Go releases

Go support is limited to official, stable Go releases compiled with the Golang toolchain.

OneAgent doesn't support binaries compiled using the gccgo toolchain.

Application binaries must be dynamically linked

This restriction applies only to Linux systems and if Go static monitoring is disabled.

OneAgent fully automatic injection requires dynamically linked application binaries. Dynamic linking is automatically applied when the application uses certain standard runtime library packages, for example, net/http.

In all other cases, you can enforce dynamic linking through the -ldflags '-linkmode=external' command line option. Note that disabling cgo, for example, using CGO_ENABLED=0, is not supported, and OneAgent will reject the resulting application binary.

Example

Consider the following minimalistic Go application called GoMinimal.go:

go
package main
import "fmt"

func main() {
    fmt.Print("Enter text: ")
    var input string
    fmt.Scanln(&input)
    fmt.Print(input)
}

Building the application results in a statically linked application binary:

shell
go build GoMinimal.go
file GoMinimal
GoMinimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

You can enforce dynamic linking with -ldflags '-linkmode=external':

shell
go build -ldflags '-linkmode=external' GoMinimal.go
file GoMinimal
GoMinimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32

Support for static monitoring

Dynatrace OneAgent supports Go static application monitoring starting with OneAgent version 1.203. For instructions on turning this feature on, see Enable Go static monitoring.

Limitations

OneAgent fully automatic static injection is supported if the parent process, for example, a shell like /bin/bash that starts the statically linked Go binary, is dynamically linked. However, after you turn on Go static monitoring, restart the parent process to enable automatic injection.

Static Go applications which use cgo are not supported

OneAgent rejects monitoring of static Go binaries that use cgo and therefore have a static dependency on C system library libc. That version of the statically linked libc may conflict with the one used by OneAgent.

Workaround

Build the Go application as dynamically linked executable (which dynamically links against libc). This makes sure that both the Go application and OneAgent use the same version of libc, the one which is available on the host.

Static Go monitoring requires the SYS_PTRACE capability in Docker containers

The SYS_PTRACE capability is enabled by default for Docker 19.03.0+ and Linux Kernel 4.8+. It allows system calls between processes running in a container, which is a requirement for Go static monitoring.

Workaround

For Docker version less than 19.03.0 or Linux Kernel version less than 4.8, run the container with the SYS_PTRACE capability:

shell
docker run --cap-add=SYS_PTRACE <container> ...

Docker images that don't provide a C system library are not supported

OneAgent requires a C system library to be available on the monitored host.

Example and workaround

An example would be the scratch image:

dockerfile
FROM scratch
COPY StaticGoMinimal /
CMD ["/StaticGoMinimal"]

To overcome this limitation, change the base image of a container to one that provides a C system library, for example, the alpine image:

dockerfile
FROM alpine:3.11
COPY StaticGoMinimal /
CMD ["/StaticGoMinimal"]

Side effects

The file proc/<pId>/exe refers to an executable named oneagentdynamizer instead of the Go application binary, it is contained in the proc pseudo-filesystem that provides an interface to kernel data structures of running processes. This may cause system tools like ps or top to display oneagentdynamizer instead of the Go binary name in their output.

Applications can't be built with -linkshared option

Go supports dynamic linking of the Go standard library. This build mode is rarely used, and OneAgent won't inject into applications built this way.

Example

Consider the following minimalistic Go application called GoMinimal.go:

shell
go install -buildmode=shared -linkshared std
go build -linkshared GoMinimal.go

OneAgent will reject the resulting application binary.

Applications that load Go plugins aren't supported

A Go plugin is a package compiled using the -buildmode=plugin build flag to produce a shared object file. This build mode is rarely used, and OneAgent will disable deep monitoring when an application actually loads a Go plugin.

Vendored third-party packages aren't supported

Go vendoring is used to include local copies of external dependencies in the project repository. This approach was used to pin versions of third-party packages before Go module support was added.

OneAgent will not monitor vendored packages. For example, gRPC services are supported only if you use Go modules or if you import go-grpc directly without using a dependency management system.

Applications must contain a symbol table

OneAgent relies on the information stored in the application binary file's symbol table. By default, Go generates a symbol table into the application binary, but this can be suppressed by command line parameters or external tools like strip.

The rarely used go run <application> command builds and runs applications on the fly. Because the output application file is temporary—the file is deleted automatically after the app termination—the application binary contains no symbol table. Thus, OneAgent cannot monitor an application that was generated with the go run <application> command.

Applications built with race detector enabled aren't supported

An application built with -race flag contains a built-in data race detector. This build mode is mostly used in a development environment and OneAgent won't inject into applications built this way.

Creation stack profiling of OS threads is disabled

OneAgent does not support the predefined threadcreate profile. Thread creation profiling results of Go applications monitored by OneAgent will contain empty stack traces only.

Support for musl libc

The musl libc library is a drop-in replacement for the glibc library. Dynatrace supports musl-based Go applications, such as those built on Alpine Linux.

There is one additional requirement for building a dynamically linked application binary. You should use the Go toolchain for alpine (golang:<version>-alpine) and add -ldflags '-linkmode external' to the build command line to enforce usage of the system linker. This is not required for statically linked Go applications watched by Go static monitoring.

Details

While musl libc does closely mimic the glibc functionalities, there are subtle behavioral differences between the two. Moreover, Go doesn't officially support the musl-based Go toolchain, which means Go toolchain binaries can't be downloaded from the golang.org website.

In addition, there is a serious issue with how Go uses musl libc. This limits the extent to which Dynatrace can support musl-based applications. The Go toolchain includes an internal linker that generates musl-based application binaries that don't correctly initialize musl libc at application startup. This issue prevents Dynatrace from monitoring these applications. In such a case, the following message is displayed on the relevant application process page:
Activation of deep monitoring was unsuccessful, Monitoring of Go musl binaries built with Go internal linker is not supported

If you use the system linker to generate the application binary, it adds startup code that correctly initializes shared objects. Also, adding -ldflags 'linkmode external' to the build command line enforces usage of the system linker. The resulting binary will execute with a correctly initialized libc, allowing Dynatrace to monitor such an application.

Example

Consider the following minimalistic Go application called GoMinimal.go:

go
package main
import "fmt"

func main() {
    fmt.Print("Enter text: ")
    var input string
    fmt.Scanln(&input)
    fmt.Print(input)
}

The following multi-stage docker file yields a valid dynamically linked Go musl binary in stage 1 and runs the application in stage 2.

dockerfile
# --- Stage 1:
# Use Golang toolchain for alpine to build the application.
FROM golang:1.13.5-alpine as builder
RUN apk update && apk add gcc libc-dev

# Copy local code, for example, GoMinimal.go, to the container image.
COPY ./GoMinimal.go ./GoMinimal.go

# Build dynamically linked Go binary.
RUN go build -ldflags '-linkmode=external' GoMinimal.go

# --- Stage 2:
# Use a Docker multi-stage build to create a lean production image.
FROM alpine:3.11
# Install ca-certificates and libc6-compat for Go programs to work properly.
RUN apk add --no-cache ca-certificates libc6-compat

# Copy the binary to the production image from the builder stage.
COPY --from=builder /go/GoMinimal /GoMinimal

# Run the application on container startup.
CMD ["/GoMinimal"]

Build the container and run the application:

shell
docker build -t gominimal-alpine .
docker run --interactive gominimal-alpine