Repository navigation
Service shutdown when instrumenting with javaagent #255
Description
Tried instrumenting one of the hypertrace java service (https://github.com/hypertrace/hypertrace-service/tree/main/hypertrace-data-config-service) with java agent.
The service was deployed in k8s cluster locally using helm.
Dockerfile used for building the image (locally built the service image to embed agent):
FROM hypertrace/javaagent:latest
ARG COMMIT_SHA=unknown
LABEL commit_sha=${COMMIT_SHA}
ENV COMMIT_SHA=${COMMIT_SHA}
LABEL maintainer="Hypertrace 'https://www.hypertrace.org/'"
#RUN wget https://github.com/hypertrace/javaagent/releases/latest/download/hypertrace-agent-all.jar -O /javaagent.jar ;
WORKDIR /app
COPY scripts/hypertrace-data-config-service run
COPY libs libs
COPY resources resources
COPY classes classes
ENV SERVICE_NAME=hypertrace-data-config-service CLUSTER_NAME=default-cluster POD_NAME=default-pod
ENTRYPOINT ["./run"]
The docker image can be found here: https://hub.docker.com/repository/docker/rishabh691/hypertrace-data-config-service (rishabh691/hypertrace-data-config-service:test1)
Following env vars were passed in k8s deployment.
- name: JAVA_TOOL_OPTIONS
value: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
- name: HT_SERVICE_NAME
value: data-config-service
- name: HT_REPORTING_ENDPOINT
value: http://hypertrace-oc-collector:9411/api/v2/spans
The service crashes with following error:
Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
[main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"data-config-service","reporting":{"endpoint":"http://hypertrace-oc-collector:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-07 12:59:11:966 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.9.7
[opentelemetry.auto.trace 2021-02-07 12:59:12:169 +0000] [main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"data-config-service","reporting":{"endpoint":"http://hypertrace-oc-collector:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-07 12:59:12:583 +0000] [main] INFO io.opentelemetry.javaagent.tooling.PropagatorsInitializer - Added tracecontext propagator
[opentelemetry.auto.trace 2021-02-07 12:59:14:280 +0000] [main] INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller - Installed span exporter: io.opentelemetry.exporter.zipkin.ZipkinSpanExporter
[opentelemetry.auto.trace 2021-02-07 12:59:17:082 +0000] [main] WARN io.opentelemetry.javaagent.tooling.InstrumentationModule - Helper classes and resources won't be injected if no types are instrumented: methods
Hypertrace agent started, version: 0.9.7
2021-02-07 12:59:30.247 [main] INFO o.h.c.s.PlatformServiceLauncher - Trying to start PlatformService ...
2021-02-07 12:59:30.355 [main] WARN o.h.c.s.c.ConfigUtils - Cannot find Property: bootstrap.config.uri in neither JVM parameter nor OS environment
2021-02-07 12:59:30.440 [main] WARN o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-07 12:59:30.440 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-07 12:59:30.441 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
[opentelemetry.auto.trace 2021-02-07 12:59:30:833 +0000] [opentelemetry-exec-shutdown-hook] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for {}
2021-02-07 12:59:32.357 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-07 12:59:32.431 [main] INFO o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-07 12:59:32.446 [main] INFO o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-07 12:59:32.533 [main] INFO o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-07 12:59:32.534 [main] INFO o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-07 12:59:33.150 [main] WARN o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-07 12:59:33.150 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-07 12:59:33.150 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
2021-02-07 12:59:33.156 [main] INFO o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-07 12:59:33.231 [main] INFO o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-07 12:59:33.231 [main] INFO o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-07 12:59:33.231 [main] INFO o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-07 12:59:33.233 [main] INFO o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-07 12:59:33.234 [main] INFO o.h.c.s.PlatformService - Starting the service by using this metrics configuration Config(SimpleConfigObject({}))
2021-02-07 12:59:34.143 [main] INFO o.h.c.s.m.PlatformMetricsRegistry - Trying to init PrometheusReporter
2021-02-07 12:59:34.547 [main] INFO o.h.c.s.m.PlatformMetricsRegistry - Setting default tags for all metrics to: {app=hypertrace-data-config-service}
[opentelemetry.auto.trace 2021-02-06 19:35:08:046 +0000] [Thread-0] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for {}
2021-02-06 19:35:08.133 [Thread-0] ERROR o.h.c.s.PlatformServiceLauncher - Error while calling quitquitquit
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:15020 [/127.0.0.1] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
Am I misconfiguring the agent or missing something?
Activity
pavolloffay commentedon Feb 8, 2021
I am not able to reproduce this locally
findingrish commentedon Feb 9, 2021
attaching deployment manifest to bring up the container in a Pod.