Upgrade to v0.55 from earlier versions

This setup assumes you've installed SigNoz version >=0.53 && <v0.55 and have kept it running for at least 15 days (the default retention period).

If you changed the default retention period, please make the adjustments outlined in the following step before upgrading to v0.55.

You can remove these flags after the system has run for the duration specified in your retention settings.

Please update the clickhouse queries in dashboards and alerts by following the guide here.

Config to Keep Using Old Table

For Docker

• In your docker-compose.yaml, add the following:

  query-service:
    command:
      [
        "-config=/root/config/prometheus.yml",
        "--use-logs-new-schema=false"
      ]
  

• In your otel-collector-config.yaml, add the following:

  exporters:
    clickhouselogsexporter:
      use_new_schema: false
  

For Kubernetes

• In your override-values.yaml, add the following:

  queryService:
    additionalArgs:
      - --use-logs-new-schema=false
  otelCollector:
    config:
      exporters:
        clickhouselogsexporter:
          use_new_schema: false
  

Migrating the Existing Materialized Columns and TTL Settings

After upgrading to SigNoz version v0.55 i.e. SigNoz chart version v0.53.1, you need to run the migration script to copy the TTL settings and materialized columns from the old table.

Migration changes include:

• materialized columns from the old table to the new table
• TTL settings from the old table to the new table

Steps to Run Migration Script

First Upgrade to v0.55

Follow the platform specific instructions to upgrade to 0.55 and above.

• Docker Standalone
• Kubernetes

For Docker

docker run --name signoz-migrate-55 --network clickhouse-setup_default \
  -it -d signoz/migrate:0.55 -host=clickhouse -port=9000

Steps to check logs:

docker logs -f signoz-migrate-55

In case of failure and have to run again, make sure to cleanup the container before running the migration script again.

docker stop signoz-migrate-55

docker rm signoz-migrate-55

For Kubernetes

RELEASE=my-release
NAMESPACE=platform
ADMIN_PASSWORD=$(
  kubectl -n $NAMESPACE get clickhouseinstallations.clickhouse.altinity.com $RELEASE-clickhouse \
  -o jsonpath --template '{.spec.configuration.users.admin/password}'
)

kubectl -n $NAMESPACE run -i -t signoz-migrate-55 --image=signoz/migrate:0.55 --restart='Never' \
  -- -host=$RELEASE-clickhouse -port=9000 -userName=admin -password=$ADMIN_PASSWORD

Steps to check logs:

kubectl -n $NAMESPACE logs -f signoz-migrate-55

In case of failure and have to run again, make sure to cleanup the pod before running the migration script again.

kubectl -n $NAMESPACE delete pod signoz-migrate-55

In case of Upgrade Failure

If you face any issue, reach out to us at Slack.

Command-Line Interface (CLI) Flags

There are some custom flags which can be enabled based on different use-cases. All the flags below are optional.

Flags:

• -port : Specify port of clickhouse. default=9000
• -host : Specify host of clickhouse. default=127.0.0.1
• -userName : Specify user name of clickhouse. default=default
• -password : Specify password of clickhouse. default=""

Migrate Data from Old Table to New Table

If you already kept SigNoz version >=0.53 && <v0.55 running for at-least 15 days i.e the default retention period this is not required as data will already be backfilled.

This guide will help migrate the data from old table to new table.

For migrating the data you will need to get the start and end timestamp which is the duration of logs data that you want to migrate from the old table to the new table.

For Docker

• Exec into clickhouse container

  docker exec -it signoz-clickhouse /bin/bash
  
  clickhouse client
  

• Get the starting timestamp

  select timestamp from signoz_logs.distributed_logs order by timestamp asc limit 1
  

• Get the ending timestamp

  select timestamp from signoz_logs.distributed_logs_v2 order by timestamp asc limit 1
  

  In case of empty response from the above query, please make sure that you upgraded to version >=0.53 first.

For Kubernetes

• Exec into your clickhouse container

  kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh
  
  clickhouse client
  

• Get the starting timestamp

  select timestamp from signoz_logs.distributed_logs order by timestamp asc limit 1
  

• Get the ending timestamp

  select timestamp from signoz_logs.distributed_logs_v2 order by timestamp asc limit 1
  

  In case of empty response from the above query, please make sure that you upgraded to version >=0.53 first.

Next we will run the migrator with arguments that will copy the data from the old to the new table.

For Docker

docker run --name signoz-migrate-55-data --network clickhouse-setup_default \
  -it -d signoz/migrate:0.55-data -host=clickhouse -port=9000 -dest_host=clickhouse -dest_port=9000 -start_ts=<START_TIMESTAMP> -end_ts=<END_TIMESTAMP> -batch_size=40000 -batch_duration=5

Please replace <START_TIMESTAMP> and <END_TIMESTAMP>

Steps to check logs:

docker logs -f signoz-migrate-55-data

In case of any failure, you have to run the script again with updated ending timestamp.

• Get the updated ending timestamp

  select timestamp from signoz_logs.distributed_logs_v2 order by timestamp asc limit 1
  

• Make sure to cleanup the container before running the migration script again.

  docker stop signoz-migrate-55-data
  
  docker rm signoz-migrate-55-data
  

For Kubernetes

• Create a pod.yaml file

apiVersion: v1
kind: Pod
metadata:
  name: signoz-migrate-data
spec:
  restartPolicy: Never
  containers:
    - name: signoz-migrate-data
      image: signoz/migrate:0.55-data
      args:
        - "-host=<RELEASE>-clickhouse"
        - "-dest_host=<RELEASE>-clickhouse"
        - "-username=admin"
        - "-dest_username=admin"
        - "-password=<ADMIN_PASSWORD>"
        - "-dest_password=<ADMIN_PASSWORD>"
        - "-start_ts=<START_TIMESTAMP>"
        - "-end_ts=<END_TIMESTAMP>"
        - "-batch_size=40000"
        - "-batch_duration=5"
      resources:
        requests:
          cpu: 1000m
          memory: 1Gi

Please replace <RELEASE>, <ADMIN_PASSWORD>, <START_TIMESTAMP> and <END_TIMESTAMP>

To get the <ADMIN_PASSWORD>

RELEASE=my-release
NAMESPACE=platform
ADMIN_PASSWORD=$(
  kubectl -n $NAMESPACE get clickhouseinstallations.clickhouse.altinity.com $RELEASE-clickhouse \
  -o jsonpath --template '{.spec.configuration.users.admin/password}'
)

echo $<ADMIN_PASSWORD>

Now, create the signoz-migrate-data pod using the YAML:

kubectl -n <NAMESPACE> apply -f pod.yaml

Steps to check logs:

kubectl -n $NAMESPACE logs -f signoz-migrate-data

In case of any failure, you have to run the script again with updated ending timestamp.

• Get the updated ending timestamp

  select timestamp from signoz_logs.distributed_logs_v2 order by timestamp asc limit 1
  

• Make sure to cleanup the container before running the migration script again.

  kubectl -n $NAMESPACE delete pod signoz-migrate-data
  

In case of Upgrade Failure

If you face any issue, reach out to us on Slack.

Data Migrator Command-Line Interface (CLI) Flags

There are the flags for data migrator which can be customized based on different use-cases.

Flags:

• -port : Specify port of clickhouse. default=9000
• -host : Specify host of clickhouse. default=127.0.0.1
• -username : Specify user name of clickhouse. default=default
• -password : Specify password of clickhouse. default=""
• -dest_port : Specify dest port of clickhouse. default=9000
• -dest_host : Specify dest host of clickhouse. default=127.0.0.1
• -dest_username : Specify dest user name of clickhouse. default=default
• -dest_password : Specify dest password of clickhouse. default=""
• -start_ts : Specify starting timestamp for the logs data to be migrated
• -end_ts : Specify ending timestamp for the logs data to be migrated
• -batch_duration : Specify window size of the data that will be fetched from the old table
• -batch_size : Specify size of the batches that will be written to the new table