Blogpost

Passage of time

1 minute read

Fixing abnormally high CPU consumption in Golang

Find out why your Golang code is consuming abnormally high CPU.

At DeltaBlue we have been using Golang to create highly scalable and performant components with a very specific function in an application architecture, such as a data router to forward IOT data to a specific hub, a regulator to throttle and forward data harvesting requests or a websocket server with dedicated channels.

Recently one of our binaries was taking up large amounts of CPU usage without processing actual requests. When it had been running for a while it would consume 80% of CPU resources even when it was functionally idling.

The team got together and we soon found out that there were a lot of hanging threads, or processes that were once started but never stopped.

The first thing we needed in order to find a solution, was a script that monitored how many active processes the code was actually generating. To do this we used the UNIX command LSOF, meaning “LIST OPEN FILES”. For ease of development we wrote this script:

#!/bin/bash
OLD=""
while true
    do
        PID="$(pidof bin/server)"
        if [[ $PID != $OLD ]]; then
            echo ""
            echo "   New process started."
            echo ""
        fi
        count="$(lsof -n | grep $PID | grep -i tcp | wc -l)"
        echo "$count"
        OLD="$PID"
        sleep 2s
done

This script will fetch all processes opened by “bin/server”. Normal and expected behaviour is:

  • your code starts with X processes
  • process count increases when actions are performed
  • process count drops back to X when idle state is reached again

If that is not the case this means that the code starts processes but does not close them, which in time will lead to severe performance decrease.

Happy debugging!