Get the latest tech news
Understanding Airflow DAG and Task Concurrency on Google Cloud Composer
Learn about Airflow concurrency across Cloud Composer, installation, Directed Acyclic Graph (DAG) and task concurrency.
Setting[celery]worker_concurrency to a high value means that every worker can pick up a lot of tasks, so under certain circumstances the queue might never fill up, causing autoscaling to never trigger. Scenario: Composer Environment frequently reaches maximum limits for workers, number of tasks in queue is consistently high, and DAGs do not meet their SLAs. By understanding the core concepts, configuring your environment effectively, and employing practical optimization strategies, you can orchestrate even the most complex data pipelines with confidence.
Or read this on Hacker News