T282033: AirFlow collaboration between PE and DE

Phabricator Link	T282033
Status	open
Priority	High (red)
Points
Phabricator Task Created	2021/05/05 02:10 PM
Wiki Page Created	2021/05/05 03:44 PM
Phabricator Task Last Modified	2021/05/10 12:16 PM
Wiki Page Last Updated	2022/01/11 08:54 PM
Phabricator Task Closed
Authored By	Milimetric (Dan Andreescu)
Assigned To
Projects	Airflow (Epics) Data-Engineering (Transform) Epic Platform Team Workboards (Image Suggestion API) (Backlog)
Platform Engineering Initiative Column
Platform Team Workboards Column	Image Suggestion API: Backlog
Subtasks	T270613: [SPIKE] can we orchestrate the Similarusers pipeline with airflow? (resolved) T280799: [SPIKE] 📊Research how to run image matching on airflow (open) T282035: Catalog, Categorize, and Templetize existing scheduled workflows (open) T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance (resolved) T284225: Create airflow instances for Platform Engineering and Research (resolved) T271429: Replace Oozie with better workflow scheduler (open) T284566: Replace Airflow's HDFS client (snakebite) with pyarrow (open) T288247: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? (open) T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors (declined) T290664: Agree on a repository structure for Airflow-related code (duplicate) T290918: hdfs directory for analytics-research (resolved) T294026: [Airflow] Create repository for Airflow DAGs (open) T294024: [Airflow] Automate sync'ing archiva packages to HDFS (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇒ Incoming	5 May 2021
⇑ Analytics		5 May 2021
⇑ Platform Engineering		5 May 2021
⇓ Platform Engineering		5 May 2021
⇑ Platform Team Workboards (Image Suggestion API)		5 May 2021
Analytics	⇐ Incoming	6 May 2021
Analytics	⇒ Smart Tools for Better Data	6 May 2021
Analytics	⇐ Smart Tools for Better Data	10 May 2021
Analytics	⇒ Smart Tools for Better Data	10 May 2021
Analytics	⇐ Smart Tools for Better Data	25 June 2021
Analytics	⇒ Airflow	25 June 2021
⇓ Analytics		5 November 2021
⇑ Data-Engineering		5 November 2021
⇑ Airflow		5 November 2021
Airflow	⇐ Backlog	5 November 2021
Airflow	⇒ Epics	5 November 2021
Data-Engineering	⇐ Incoming	8 November 2021
Data-Engineering	⇒ Transform	8 November 2021
⇑ Epic		8 November 2021