I recently worked on a project focused on log anomaly detection using manageable machine learning pipelines. The pipelines mainly include data collection --- feature extraction --- feature engineering --- detection/prediction --- updating (maintenance). It’s important to have a solid UI to manage the pipelines so I can easily review the chain of pipelines. After much research, I found many engineers recommended Airflow. In airflow, the core concept is the Directed Acyclic Graph (DAG). Through the implementation, I have confirmed that this is a truly powerful tool to manage the machine learning pipelines, instead of relying on shell scripts. But, I did encounter some challenges during the process and also, fortunately, found solutions for them. The challenges can be split into two main aspects, pipeline management and dynamic generation for tasks. Pipeline...
Implementation and Dynamic Generation for Tasks in Apache Airflow
read more
