Difference between revisions of "T282033: Airflow collaborations"

Latest revision as of 20:54, 11 January 2022

Phabricator Link	T282033
Status	open
Priority	High (red)
Points
Phabricator Task Created	2021/05/05 02:10 PM
Wiki Page Created	2021/05/05 03:44 PM
Phabricator Task Last Modified	2022/01/11 07:32 PM
Wiki Page Last Updated	2022/01/11 08:54 PM
Phabricator Task Closed
Authored By	Milimetric (Dan Andreescu)
Assigned To
Projects	Airflow (Epics) Data-Engineering (Transform) Epic Platform Team Workboards (Image Suggestion API) (Backlog)
Platform Engineering Initiative Column
Platform Team Workboards Column	Image Suggestion API: Backlog
Subtasks	T270613: [SPIKE] can we orchestrate the Similarusers pipeline with airflow? (resolved) T280799: [SPIKE] 📊Research how to run image matching on airflow (open) T282035: Catalog, Categorize, and Templetize existing scheduled workflows (open) T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance (resolved) T284225: Create airflow instances for Platform Engineering and Research (resolved) T271429: Replace Oozie with better workflow scheduler (open) T284566: Replace Airflow's HDFS client (snakebite) with pyarrow (open) T288247: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? (open) T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors (declined) T290664: Agree on a repository structure for Airflow-related code (duplicate) T290918: hdfs directory for analytics-research (resolved) T294026: [Airflow] Create repository for Airflow DAGs (open) T294024: [Airflow] Automate sync'ing archiva packages to HDFS (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇒ Incoming	5 May 2021
⇑ Analytics		5 May 2021
⇑ Platform Engineering		5 May 2021
⇓ Platform Engineering		5 May 2021
⇑ Platform Team Workboards (Image Suggestion API)		5 May 2021
Analytics	⇐ Incoming	6 May 2021
Analytics	⇒ Smart Tools for Better Data	6 May 2021
Analytics	⇐ Smart Tools for Better Data	10 May 2021
Analytics	⇒ Smart Tools for Better Data	10 May 2021
Analytics	⇐ Smart Tools for Better Data	25 June 2021
Analytics	⇒ Airflow	25 June 2021
⇓ Analytics		5 November 2021
⇑ Airflow		5 November 2021
⇑ Data-Engineering		5 November 2021
Airflow	⇐ Backlog	5 November 2021
Airflow	⇒ Epics	5 November 2021
Data-Engineering	⇐ Incoming	8 November 2021
Data-Engineering	⇒ Transform	8 November 2021
⇑ Epic		8 November 2021

@@ Line 5: / Line 5: @@
 |points=
 |dateCreated=1620238223
-|dateModified=1631199916
+|dateModified=1641947562
 |author={{Phabricator User
 |type=author
@@ Line 12: / Line 12: @@
 }}
 |projects={{Phabricator Project
-|name=Analytics
+|name=Airflow
-|column=Airflow
+|column=Epics
-|entryDate=1624636194
+|entryDate=1636146613
+}}{{Phabricator Project
+|name=Data-Engineering
+|column=Transform
+|entryDate=1636390643
+}}{{Phabricator Project
+|name=Epic
+|entryDate=1620238223
 }}{{Phabricator Project
 |name=Platform Team Workboards (Image Suggestion API)
@@ Line 20: / Line 27: @@
 |entryDate=1620238223
 }}
-|subtasks=288247,280799,270613,290664,288271,284225,282035,272973,271429,284566
+|subtasks=290918,288247,280799,270613,294026,294024,290664,288271,284225,272973,271429,284566,282035
 |transitions={{Transition
+|date=1636390668
+|type=Added Project
+|project=Epic
+}}{{Transition
+|date=1636390643
+|type=Entered Column
+|project=Data-Engineering
+|column=Transform
+}}{{Transition
+|date=1636390643
+|type=Exited Column
+|project=Data-Engineering
+|column=Incoming
+}}{{Transition
+|date=1636146613
+|type=Entered Column
+|project=Airflow
+|column=Epics
+}}{{Transition
+|date=1636146613
+|type=Exited Column
+|project=Airflow
+|column=Backlog
+}}{{Transition
+|date=1636127602
+|type=Removed Project
+|project=Analytics
+}}{{Transition
+|date=1636127602
+|type=Added Project
+|project=Data-Engineering
+}}{{Transition
+|date=1636127602
+|type=Added Project
+|project=Airflow
+}}{{Transition
 |date=1624636194
 |type=Entered Column