At DeltaBlue we have been using Golang to create highly scalable and performant components with a very specific function in an application architecture, such as a data router to forward IOT data to a specific hub, a regulator to throttle and forward data harvesting requests or a websocket server with dedicated channels.
Recently one of our binaries was taking up large amounts of CPU usage without processing actual requests. When it had been running for a while it would consume 80% of CPU resources even when it was functionally idling.
The team got together and we soon found out that there were a lot of hanging threads, or processes that were once started but never stopped.
The first thing we needed in order to find a solution, was a script that monitored how many active processes the code was actually generating. To do this we used the UNIX command LSOF, meaning “LIST OPEN FILES”. For ease of development we wrote this script:
#!/bin/bash
OLD=""
while true
do
PID="$(pidof bin/server)"
if [[ $PID != $OLD ]]; then
echo ""
echo " New process started."
echo ""
fi
count="$(lsof -n | grep $PID | grep -i tcp | wc -l)"
echo "$count"
OLD="$PID"
sleep 2s
done
This script will fetch all processes opened by “bin/server”. Normal and expected behaviour is:
- your code starts with X processes
- process count increases when actions are performed
- process count drops back to X when idle state is reached again
If that is not the case this means that the code starts processes but does not close them, which in time will lead to severe performance decrease.
Happy debugging!