Difference between revisions of "T282033: Airflow collaborations"

Revision as of 14:53, 8 November 2021

Phabricator Link	T282033
Status	open
Priority	High (red)
Points
Phabricator Task Created	2021/05/05 02:10 PM
Wiki Page Created	2021/05/05 03:44 PM
Phabricator Task Last Modified	2021/11/08 11:57 AM
Wiki Page Last Updated	2022/01/11 08:54 PM
Phabricator Task Closed
Authored By	Milimetric (Dan Andreescu)
Assigned To
Projects	Airflow (Epics) Data-Engineering (Transform) Epic Platform Team Workboards (Image Suggestion API) (Backlog)
Platform Engineering Initiative Column
Platform Team Workboards Column	Image Suggestion API: Backlog
Subtasks	T270613: [SPIKE] can we orchestrate the Similarusers pipeline with airflow? (resolved) T280799: [SPIKE] 📊Research how to run image matching on airflow (open) T282035: Catalog, Categorize, and Templetize existing scheduled workflows (open) T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance (resolved) T284225: Create airflow instances for Platform Engineering and Research (resolved) T271429: Replace Oozie with better workflow scheduler (open) T284566: Replace Airflow's HDFS client (snakebite) with pyarrow (open) T288247: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? (open) T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors (declined) T290664: Agree on a repository structure for Airflow-related code (duplicate) T290918: hdfs directory for analytics-research (resolved) T294026: [Airflow] Create repository for Airflow DAGs (open) T294024: [Airflow] Automate sync'ing archiva packages to HDFS (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇒ Incoming	5 May 2021
⇑ Analytics		5 May 2021
⇑ Platform Engineering		5 May 2021
⇓ Platform Engineering		5 May 2021
⇑ Platform Team Workboards (Image Suggestion API)		5 May 2021
Analytics	⇐ Incoming	6 May 2021
Analytics	⇒ Smart Tools for Better Data	6 May 2021
Analytics	⇐ Smart Tools for Better Data	10 May 2021
Analytics	⇒ Smart Tools for Better Data	10 May 2021
Analytics	⇐ Smart Tools for Better Data	25 June 2021
Analytics	⇒ Airflow	25 June 2021
⇓ Analytics		5 November 2021
⇑ Airflow		5 November 2021
⇑ Data-Engineering		5 November 2021
Airflow	⇐ Backlog	5 November 2021
Airflow	⇒ Epics	5 November 2021
Data-Engineering	⇐ Incoming	8 November 2021
Data-Engineering	⇒ Transform	8 November 2021
⇑ Epic		8 November 2021

@@ Line 5: / Line 5: @@
 |points=
 |dateCreated=1620238223
-|dateModified=1636146613
+|dateModified=1636390668
 |author={{Phabricator User
 |type=author
@@ Line 17: / Line 17: @@
 }}{{Phabricator Project
 |name=Data-Engineering
-|column=Incoming
+|column=Transform
+|entryDate=1636390643
+}}{{Phabricator Project
+|name=Epic
 |entryDate=1620238223
 }}{{Phabricator Project
@@ Line 26: / Line 29: @@
 |subtasks=290918,288247,280799,270613,294026,294024,290664,288271,284225,272973,271429,284566,282035
 |transitions={{Transition
+|date=1636390668
+|type=Added Project
+|project=Epic
+}}{{Transition
+|date=1636390643
+|type=Entered Column
+|project=Data-Engineering
+|column=Transform
+}}{{Transition
+|date=1636390643
+|type=Exited Column
+|project=Data-Engineering
+|column=Incoming
+}}{{Transition
 |date=1636146613
 |type=Entered Column