T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events

Phabricator Link	T251609
Status	open
Priority	High (red)
Points
Phabricator Task Created	2020/05/01 11:41 AM
Wiki Page Created	2020/05/01 12:19 PM
Phabricator Task Last Modified	2020/10/21 11:14 AM
Wiki Page Last Updated	2021/10/29 02:09 PM
Phabricator Task Closed
Authored By	Ottomata (Andrew Otto)
Assigned To	Ottomata (Andrew Otto)
Projects	Analytics (Event Platform) Analytics-EventLogging (Backlog) Analytics-Kanban Event-Platform (In Progress) MW-1.35-notes (1.35.0-wmf.41; 2020-07-14) MW-1.36-notes (1.36.0-wmf.8; 2020-09-08) Patch-For-Review
Platform Engineering Initiative Column
Platform Team Workboards Column
Subtasks	T242454: Add examples to all event schemas (resolved) T251935: All EventGate instances should use EventStreamConfig (resolved) T253157: eventgate-wikimedia should expose runtime stream configuration (resolved) T250844: MEP: canary events so we know events are flowing through pipeline (duplicate) T252585: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete (open) T259898: Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source (resolved) T260375: Integrate Wikimedia Event Utilities with discovery-parent-pom (resolved) T261356: Allow analytics VLAN to reach all eventgate instances in all DCs (resolved) T263696: Q2 goal. Deploy the canary event monitoring for some event streams (resolved) T266798: Enable canary events for all streams (open) T267073: Exclude canary events from refined event Hive tables (declined)
Parent Tasks	T238230: Decommission EventLogging backend components by migrating to MEP

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
⇓ Patch-For-Review		1 May 2020
⇑ Analytics		1 May 2020
⇑ Patch-For-Review		1 May 2020
⇑ Analytics-Kanban		1 May 2020
⇑ Event-Platform		1 May 2020
⇑ Analytics-EventLogging		1 May 2020
Event-Platform	⇐ Backlog	1 May 2020
Event-Platform	⇒ In Progress	1 May 2020
Analytics-Kanban	⇐ Next Up	4 May 2020
Analytics-Kanban	⇒ In Progress	4 May 2020
⇑ Patch-For-Review		5 May 2020
Analytics	⇐ Incoming	11 May 2020
Analytics	⇒ Event Platform	11 May 2020
Event-Platform	⇐ In Progress	14 May 2020
Event-Platform	⇒ Next Up	14 May 2020
⇑ MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)		30 May 2020
⇓ MW-1.35-notes (1.35.0-wmf.35; 2020-06-02)		13 July 2020
⇑ MW-1.35-notes (1.35.0-wmf.41; 2020-07-14)		13 July 2020
Event-Platform	⇐ Next Up	25 August 2020
Event-Platform	⇒ In Progress	25 August 2020
Analytics-Kanban	⇐ In Progress	2 September 2020
Analytics-Kanban	⇒ Ready to Deploy	2 September 2020
Analytics-Kanban	⇐ Ready to Deploy	9 September 2020
Analytics-Kanban	⇒ In Code Review	9 September 2020
⇑ MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)		9 September 2020
⇓ MW-1.36-notes (1.36.0-wmf.9; 2020-09-15)		10 September 2020
⇑ MW-1.36-notes (1.36.0-wmf.8; 2020-09-08)		10 September 2020
Analytics-Kanban	⇐ In Code Review	15 October 2020
Analytics-Kanban	⇒ In Code Review	15 October 2020
Analytics-Kanban	⇐ In Code Review	22 October 2020
Analytics-Kanban	⇒ Done	22 October 2020
Analytics-Kanban	⇐ Done	29 October 2020
Analytics-Kanban	⇒ Done	29 October 2020