Difference between revisions of "T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events"

Latest revision as of 13:09, 29 October 2021

Phabricator Link	T251609
Status	resolved
Priority	High (red)
Points
Phabricator Task Created	2020/05/01 11:41 AM
Wiki Page Created	2020/05/01 12:19 PM
Phabricator Task Last Modified	2021/04/26 07:30 PM
Wiki Page Last Updated	2021/10/29 02:09 PM
Phabricator Task Closed	2020/10/29 05:23 PM
Authored By	Ottomata (Andrew Otto)
Assigned To	Ottomata (Andrew Otto)
Projects	Analytics (Event Platform) Analytics-EventLogging (Backlog) Analytics-Kanban Event-Platform (In Progress) MW-1.35-notes (1.35.0-wmf.41; 2020-07-14) MW-1.36-notes (1.36.0-wmf.8; 2020-09-08) Patch-For-Review
Platform Engineering Initiative Column
Platform Team Workboards Column
Subtasks	T242454: Add examples to all event schemas (resolved) T251935: All EventGate instances should use EventStreamConfig (resolved) T253157: eventgate-wikimedia should expose runtime stream configuration (resolved) T250844: MEP: canary events so we know events are flowing through pipeline (duplicate) T252585: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete (open) T259898: Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source (resolved) T260375: Integrate Wikimedia Event Utilities with discovery-parent-pom (resolved) T261356: Allow analytics VLAN to reach all eventgate instances in all DCs (resolved) T263696: Q2 goal. Deploy the canary event monitoring for some event streams (resolved) T266798: Enable canary events for all streams (open) T267073: Exclude canary events from refined event Hive tables (declined)
Parent Tasks	T238230: Decommission EventLogging backend components by migrating to MEP

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
⇓ Patch-For-Review		1 May 2020
⇑ Patch-For-Review		1 May 2020
⇑ Analytics-Kanban		1 May 2020
⇑ Event-Platform		1 May 2020
⇑ Analytics-EventLogging		1 May 2020
⇑ Analytics		1 May 2020
Event-Platform	⇐ Backlog	1 May 2020
Event-Platform	⇒ In Progress	1 May 2020
Analytics-Kanban	⇐ Next Up	4 May 2020
Analytics-Kanban	⇒ In Progress	4 May 2020
⇑ Patch-For-Review		5 May 2020
Analytics	⇐ Incoming	11 May 2020
Analytics	⇒ Event Platform	11 May 2020
Event-Platform	⇐ In Progress	14 May 2020
Event-Platform	⇒ Next Up	14 May 2020
⇑ MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)		30 May 2020
⇓ MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)		13 July 2020
⇑ MW-1.35-notes (1.35.0-wmf.41; 2020-07-14)		13 July 2020
Event-Platform	⇐ Next Up	25 August 2020
Event-Platform	⇒ In Progress	25 August 2020
Analytics-Kanban	⇐ In Progress	2 September 2020
Analytics-Kanban	⇒ Ready to Deploy	2 September 2020
Analytics-Kanban	⇐ Ready to Deploy	9 September 2020
Analytics-Kanban	⇒ In Code Review	9 September 2020
⇑ MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)		9 September 2020
⇓ MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)		10 September 2020
⇑ MW-1.36-notes (1.36.0-wmf.8; 2020-09-08)		10 September 2020
Analytics-Kanban	⇐ In Code Review	15 October 2020
Analytics-Kanban	⇒ In Code Review	15 October 2020
Analytics-Kanban	⇐ In Code Review	22 October 2020
Analytics-Kanban	⇒ Done	22 October 2020
Analytics-Kanban	⇐ Done	29 October 2020
Analytics-Kanban	⇒ Done	29 October 2020

@@ Line 1: / Line 1: @@
 {{Phabricator Task
-|name=Automate ingestion and refinement into Hive of event data from Kafka
+|name=Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events
-|status=open
+|status=resolved
 |color=red
 |points=
 |dateCreated=1588347702
-|dateModified=1588710224
+|dateModified=1619479826
+|dateClosed=1604006580
 |author={{Phabricator User
 |type=author
@@ Line 17: / Line 18: @@
 }}
 |projects={{Phabricator Project
+|name=MW-1.35-notes
+|column=1.35.0-wmf.41; 2020-07-14
+|entryDate=1588347702
+}}{{Phabricator Project
+|name=MW-1.36-notes
+|column=1.36.0-wmf.8; 2020-09-08
+|entryDate=1588347702
+}}{{Phabricator Project
 |name=Analytics-EventLogging
 |column=Backlog
@@ Line 22: / Line 31: @@
 }}{{Phabricator Project
 |name=Analytics-Kanban
-|column=In Progress
+|entryDate=1588347702
-|entryDate=1588603411
 }}{{Phabricator Project
 |name=Analytics
-|column=Incoming
+|column=Event Platform
-|entryDate=1588347702
+|entryDate=1589212280
 }}{{Phabricator Project
 |name=Event-Platform
 |column=In Progress
-|entryDate=1588348875
+|entryDate=1598371567
 }}{{Phabricator Project
 |name=Patch-For-Review
 |entryDate=1588347702
 }}
-|subtasks=251935
+|subtasks=267073,261356,260375,242454,263696,259898,253157,251935,250844,266798,252585
 |transitions={{Transition
+|date=1603987935
+|type=Entered Column
+|project=Analytics-Kanban
+|column=Done
+}}{{Transition
+|date=1603987935
+|type=Exited Column
+|project=Analytics-Kanban
+|column=Done
+}}{{Transition
+|date=1603376418
+|type=Entered Column
+|project=Analytics-Kanban
+|column=Done
+}}{{Transition
+|date=1603376418
+|type=Exited Column
+|project=Analytics-Kanban
+|column=In Code Review
+}}{{Transition
+|date=1602787844
+|type=Entered Column
+|project=Analytics-Kanban
+|column=In Code Review
+}}{{Transition
+|date=1602787844
+|type=Exited Column
+|project=Analytics-Kanban
+|column=In Code Review
+}}{{Transition
+|date=1599764440
+|type=Removed Project
+|project=MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)
+}}{{Transition
+|date=1599764440
+|type=Added Project
+|project=MW-1.36-notes (1.36.0-wmf.8; 2020-09-08)
+}}{{Transition
+|date=1599674423
+|type=Added Project
+|project=MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)
+}}{{Transition
+|date=1599663533
+|type=Entered Column
+|project=Analytics-Kanban
+|column=In Code Review
+}}{{Transition
+|date=1599663533
+|type=Exited Column
+|project=Analytics-Kanban
+|column=Ready to Deploy
+}}{{Transition
+|date=1599076280
+|type=Entered Column
+|project=Analytics-Kanban
+|column=Ready to Deploy
+}}{{Transition
+|date=1599076280
+|type=Exited Column
+|project=Analytics-Kanban
+|column=In Progress
+}}{{Transition
+|date=1598371567
+|type=Entered Column
+|project=Event-Platform
+|column=In Progress
+}}{{Transition
+|date=1598371567
+|type=Exited Column
+|project=Event-Platform
+|column=Next Up
+}}{{Transition
+|date=1594674042
+|type=Removed Project
+|project=MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)
+}}{{Transition
+|date=1594674042
+|type=Added Project
+|project=MW-1.35-notes (1.35.0-wmf.41; 2020-07-14)
+}}{{Transition
+|date=1590825647
+|type=Added Project
+|project=MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)
+}}{{Transition
+|date=1589466512
+|type=Entered Column
+|project=Event-Platform
+|column=Next Up
+}}{{Transition
+|date=1589466512
+|type=Exited Column
+|project=Event-Platform
+|column=In Progress
+}}{{Transition
+|date=1589212280
+|type=Entered Column
+|project=Analytics
+|column=Event Platform
+}}{{Transition
+|date=1589212280
+|type=Exited Column
+|project=Analytics
+|column=Incoming
+}}{{Transition
 |date=1588709326
 |type=Added Project