Get the latest tech news
Debugging audio artifacts caused by... a serial port?
The joys of scaling a 10,000 node cluster At Recall.ai we run enormous infrastructure to process millions of meetings per month, in real-time. At peak load, our cluster contains over 10,000 VMs at processing over 1TB / sec of raw video.
During 2023 we grew extremely rapidly and EKS's control plane could not keep up with our spiky load patterns, so we migrated off Kubernetes and wrote our own cluster scaler to replace it. Because our customers do significantly more meetings on weekdays, we time our major deploys for Friday nights so we can monitor performance during the low-utilization periods on the weekend. I bisected the config options, and through a trial-and-error-binary-search determined that enabling real time transcription was the critical factor in causing these artifacts to appear.
Or read this on Hacker News