Tasks By Project

Project: Data-Engineering

Switch to Active Tasks 38 Phabricator task(s).

Phabricator Link	Wiki Link	Status	Priority	Projects	Subtasks	Parent Tasks
T120242	T120242: Consistent MediaWiki state change events \| MediaWiki events as source of truth	open	Medium (orange)	Event-Platform (Radar) DBA (Blocked) Platform Team Legacy (Later) Services (later) Analytics (Event Platform) Data-Engineering (Ingest) WMF-Architecture-Team (0 Pre-Initiative)	T215001: Revisions missing from mediawiki_revision_create	T201068: Modern Event Platform: Stream Intake Service
T216492	T216492: Page-links-change stream doesn't capture duplicated links	open	Medium (orange)	Analytics (Incoming) Data-Engineering (Instrument) Internet-Archive (Links) Event-Platform (Backlog) The-Wikipedia-Library (Backlog)		T115119: Create a feed or log of changed links on Wikimedia projects
T216504	T216504: page-links-change stream is assigning template propagation events to the wrong edits	open	Medium (orange)	Analytics (Incoming) Patch-For-Review Event-Platform (Backlog) The-Wikipedia-Library (Backlog) Data-Engineering-Kanban (Next Up) Data-Engineering (Instrument) Internet-Archive (Links) Platform Team Workboards (Clinic Duty Team) (Later)		T115119: Create a feed or log of changed links on Wikimedia projects
T217271	T217271: Some event data (like the one that comes from mediawiki events such us revision create) should not get sanitized	open	Medium (orange)	Data-Engineering (Datasets)
T233004	T233004: Schema changes for `cu_changes` and `cu_log` table	stalled	Medium (orange)	Patch-For-Review CheckUser (Backlog) Data-Engineering (Datasets) Schema-change (Add / Create) Platform Team Workboards (Clinic Duty Team) (External Code Review Completed)	T253796: Should CheckUser record the log_id of logged actions?	T174022: Implement multi-content revisions T174043: Deploy Multi-Content Revisions T196087: Refactored implementation of MCR page update interface T227047: Complete actor table and comment table migration
T238230	T238230: Decommission EventLogging backend components by migrating to MEP	open	Medium (orange)	Analytics-EventLogging (Backlog) Data-Engineering (Ingest) Patch-For-Review Analytics (Event Platform) Event-Platform (In Progress)	T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events T256370: Camus should look for multiple possible timestamp fields to use for hourly partitioining T259163: Migrate legacy metawiki schemas to Event Platform T259157: BUOD-KR1-Q3: Require that all new schema/instruments are created with the MEP system T259201: Prevent schema creation in meta for eventlogging schemas T265651: Disable eventlogging-valid-mixed topic T282167: Unblock Analytics Team: Drop MobileWebSearch schema	T185233: Modern Event Platform
T239591	T239591: Update mediawiki-history to use new Multi-Content-Revision tables	open	High (red)	Data-Engineering (Datasets) Platform Engineering (Tracking/Watching)
T239630	T239630: Mediaviewer views should be reworked to be an eventlogging event	open	Medium (orange)	Multimedia (Untriaged) Analytics (Event Platform) Event-Platform (Backlog) Data-Engineering (Ingest)		T241241: Produce an instrumentation event stream using new EPC and EventGate from client side browsers
T240387	T240387: MW REST API Historical Data Endpoint Needs	open	High (red)	Data-Engineering (Datasets)	T241185: Flink Spike T241184: Design Document that proposes an alternative architecture for historic data endpoints
T249755	T249755: Cassandra3 migration for Analytics AQS	open	High (red)	Data-Engineering-Kanban (Q2 Epics) Data-Engineering (Epics) Cassandra (Backlog) Epic Platform Team Workboards (Platform Engineering Reliability) (Watching) Analytics (Analytics-Clusters) (Q2 2021/2022)	T255141: Upgrade the Cassandra AQS cluster to Cassandra 3.11 T280649: Add a spark job loading Cassandra 3 T278699: AQS Cassandra driver needs to be updated T257572: Set up a testing environment for the AQS Cassandra 3 migration T289161: Update cassandra oozie jobs to load cassandra3 using Spark job T278701: Store AQS schema and grants in git T249756: Cassandra3 migration plan proposal T280155: Dual loading from Hive into old and new AQS clusters T290068: Check AQS with cassandra (serving + data) T297460: Send cassandra3 (new hosts) logs to logstash T297803: Switch over the Cassandra AQS cluster to the new hosts T298516: Investigate high levels of garbage collection on new AQS nodes
T255141	T255141: Upgrade the Cassandra AQS cluster to Cassandra 3.11	duplicate	High (red)	Platform Team Workboards (Platform Engineering Reliability) (Backlog) Analytics-Kanban Data-Engineering (Incoming) Data-Engineering-Kanban (Next Up) Analytics (Analytics-Clusters) (Q1 2021/2022)		T249755: Cassandra3 migration for Analytics AQS
T257572	T257572: Set up a testing environment for the AQS Cassandra 3 migration	open	Needs Triage (violet)	Cassandra (Backlog) Patch-For-Review Analytics (Analytics-Clusters) (Q2 2021/2022) Data-Engineering-Kanban (Next Up) Analytics-Kanban Data-Engineering (Serve)		T249755: Cassandra3 migration for Analytics AQS
T259163	T259163: Migrate legacy metawiki schemas to Event Platform	open	High (red)	Product-Data-Infrastructure (Watching) Product-Analytics (Tracking) Data-Engineering (Ingest) Analytics (Event Platform) MW-1.36-notes (1.36.0-wmf.18; 2020-11-17) Event-Platform (In Progress) Better Use Of Data (Tracking)	T238138: VirtualPageView Event Platform Migration T260582: Migrate EventLogging MediaViewer data to Event Platform T260580: Parse user agents in navtiming instead of relying on eventlogging to do it T262304: eventlogging-processor should fail to produce schemas that have been migrated to Event Platform T267353: VisualEditorFeatureUse Event Platform Migration T267352: UniversalLanguageSelector Event Platform Migration T267351: SuggestedTagsAction Event Platform Migration T267350: SpecialMuteSubmit Event Platform Migration T267349: SpecialInvestigate Event Platform Migration T267348: PrefUpdate Event Platform Migration T267347: MobileWebUIActionsTracking Event Platform Migration T267346: KaiOSAppFirstRun Event Platform Migration T267345: KaiOSAppFeedback Event Platform Migration T267344: InukaPageView Event Platform Migration T267343: EditAttemptStep Event Platform Migration T267342: DesktopWebUIActionsTracking Event Platform Migration T267341: CookieBlock Event Platform Migration T267340: AutoblockIpBlock Event Platform Migration T267333: Migrate Growth EventLogging schemas to Event Platform T268517: Migrate Anti-Harassment EventLogging schemas to Event Platform T270136: Automate deprecation of schema on metawiki after migration to Event Platform T271166: QuickSurveysResponses Event Platform Migration T271165: QuickSurveyInitiation Event Platform Migration T271164: DesktopWebUIActionsTracking Event Platform Migration T271163: TranslationRecommendation* Schemas Event Platform Migration T271162: ExternalLinksChange Event Platform Migration T271168: CentralNoticeBannerHistory and CentralNoticeImpression Event Platform Migration T271208: NavigationTiming Extension schemas Event Platform Migration T271207: EditAttemptStep Event Platform Migration T271358: QA Modern Event Platform editing schema migrations T273219: KaiOS / Inuka Event Platform client T275011: TemplateDataApi Event Platform Migration T275009: ReferencePreviewsPopups Event Platform Migration T275008: ReferencePreviewsCite Event Platform Migration T275007: ReferencePreviewsBaseline Event Platform Migration T275005: CodeMirrorUsage Event Platform Migration T275015: VisualEditorTemplateDialogUse Event Platform Migration T275014: TwoColConflictExit Event Platform Migration T275013: TwoColConflictConflict Event Platform Migration T275012: TemplateDataEditor Event Platform Migration T282012: WikipediaPortal Event Platform Migration T282140: WikidataCompletionSearchClicks Event Platform Migration T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned T282562: WMDEBanner* Event Platform Migration T282855: LandingPageImpression Event Platform Migration T284679: EditorActivation Event Platform Migration T287210: EchoMail and EchoInteraction Event Platform Migration	T238230: Decommission EventLogging backend components by migrating to MEP
T259712	T259712: Allow disabling/enabling configured streams via wgEventStreams config	open	High (red)	Analytics (Event Platform) Event-Platform (Backlog) Platform Team Initiatives (Modern Event Platform (TEC2)) Better Use Of Data (Inbox) Data-Engineering (Ingest) Product-Data-Infrastructure (Task Backlog)		T259704: BUOD-KR1-Q2: Upgrade MEP clients to full release status
T271429	T271429: Replace Oozie with better workflow scheduler	open	High (red)	Data-Engineering (Transform) Airflow (Epics) Product-Analytics (Tracking) Epic	T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance T284172: &#91 SPIKE&#93 analytics-airflow jobs development T241246: Spike: POC of refine with airflow T217059: Spike &#91 2019-2020 work&#93 Oozie Replacement. Airflow Study / Argo Study	T282033: Airflow collaborations
T282012	T282012: WikipediaPortal Event Platform Migration	open	Medium (orange)	Analytics (Event Platform) Event-Platform (Backlog) Readers-Web-Backlog (Needs Prioritization (Product)) Patch-For-Review Data-Engineering (Incoming)		T259163: Migrate legacy metawiki schemas to Event Platform
T282033	T282033: Airflow collaborations	open	High (red)	Airflow (Epics) Data-Engineering (Transform) Platform Team Workboards (Image Suggestion API) (Backlog) Epic	T270613: &#91 SPIKE&#93 can we orchestrate the Similarusers pipeline with airflow? T280799: &#91 SPIKE&#93 📊Research how to run image matching on airflow T282035: Catalog, Categorize, and Templetize existing scheduled workflows T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance T284225: Create airflow instances for Platform Engineering and Research T271429: Replace Oozie with better workflow scheduler T284566: Replace Airflow's HDFS client (snakebite) with pyarrow T288247: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors T290664: Agree on a repository structure for Airflow-related code T290918: hdfs directory for analytics-research T294026: &#91 Airflow&#93 Create repository for Airflow DAGs T294024: &#91 Airflow&#93 Automate sync'ing archiva packages to HDFS
T282035	T282035: Catalog, Categorize, and Templetize existing scheduled workflows	open	Medium (orange)	Platform Engineering (Tracking/Watching) Airflow (Backlog) Data-Engineering (Transform)		T282033: Airflow collaborations
T282131	T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned	open	High (red)	Data-Engineering (Ingest) Better Use Of Data (Inbox) Event-Platform (In Progress) Fundraising-Backlog (Blocked or not fr-tech) Data-Engineering-Kanban (In Progress) Product-Analytics (Tracking) Product-Data-Infrastructure (Inbox) Analytics-Kanban Analytics-EventLogging (Backlog)	T283551: Decide whether to keep the ExternalGuidance data stream T290303: Migrate WikibaseTermboxInteraction EventLogging Schema to new EventPlatform thingy	T259163: Migrate legacy metawiki schemas to Event Platform
T282887	T282887: Avoid accepting Kafka messages with whacky timestamps	open	Medium (orange)	Analytics (Operational Excellence) Product-Data-Infrastructure (Inbox) Discovery Event-Platform (Backlog) Platform Team Workboards (Clinic Duty Team) (Later) Data-Engineering (Ingest) SRE (Backlog)
T284566	T284566: Replace Airflow's HDFS client (snakebite) with pyarrow	open	Medium (orange)	Platform Engineering (Tracking/Watching) Airflow (Backlog) Data-Engineering (Transform)		T282033: Airflow collaborations
T285692	T285692: Write a job entirely in Airflow with spark and/or sparkSQL	open	High (red)	Patch-For-Review Airflow (Done) Data-Engineering-Kanban (Done) Data-Engineering (Transform)		T284172: [SPIKE] analytics-airflow jobs development
T288247	T288247: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines?	open	Needs Triage (violet)	Analytics-Kanban Airflow (Backlog) Data-Engineering (Ops)		T282033: Airflow collaborations
T288271	T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors	declined	High (red)	Data-Engineering (Transform) Data-Engineering-Kanban (Next Up) Airflow (Next Up)		T282033: Airflow collaborations T294024: [Airflow] Automate sync'ing archiva packages to HDFS
T289161	T289161: Update cassandra oozie jobs to load cassandra3 using Spark job	resolved	High (red)	Data-Engineering-Kanban (Next Up) Epic Cassandra (Backlog) Analytics-Kanban Analytics (Smart Tools for Better Data) Data-Engineering (Incoming)		T249755: Cassandra3 migration for Analytics AQS
T290068	T290068: Check AQS with cassandra (serving + data)	resolved	High (red)	Data-Engineering-Kanban (Next Up) Analytics (Analytics Query Service) Epic Data-Engineering (Serve) Analytics-Kanban	T291470: Repair and reload cassandra2 mediarequest_per_file data table T291469: Repair and reload all cassandra-2 data tables but the 2 big ones T291472: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances T291473: Test snapshot-reload from all instances using pageview-top data table	T249755: Cassandra3 migration for Analytics AQS
T290303	T290303: Migrate WikibaseTermboxInteraction EventLogging Schema to new EventPlatform thingy	open	High (red)	Event-Platform (Backlog) Wikidata-Campsite (Incoming) Data-Engineering (Ingest) Data-Engineering-Kanban (In Progress) Analytics (Event Platform) Wikidata (incoming) Wikidata-Termbox (Backlog) wdwb-tech (Active)		T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned
T291120	T291120: MediaWiki Events as a Source of Truth - Problem Statement	open	High (red)	Analytics (Event Platform) Data-Engineering (Ingest) Event-Platform (Backlog) tech-decision-forum (Research and Prototyping) Platform Engineering (Inbox)
T291469	T291469: Repair and reload all cassandra-2 data tables but the 2 big ones	resolved	High (red)	Analytics-Kanban Analytics (Analytics Query Service) Data-Engineering (Incoming)		T290068: Check AQS with cassandra (serving + data)
T291470	T291470: Repair and reload cassandra2 mediarequest_per_file data table	resolved	High (red)	Data-Engineering (Serve) Data-Engineering-Kanban (Done) Analytics-Kanban		T290068: Check AQS with cassandra (serving + data)
T291472	T291472: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances	resolved	High (red)	Analytics-Kanban Analytics (Analytics Query Service) Data-Engineering (Serve) Data-Engineering-Kanban (Done)		T290068: Check AQS with cassandra (serving + data) T291473: Test snapshot-reload from all instances using pageview-top data table
T291620	T291620: Better observability/visualization for MediaWiki jobs	open	Low (yellow)	Data-Engineering (Datasets) Wikibase change dispatching scripts to jobs (Later/External) Platform Team Workboards (Platform Engineering Reliability) (Backlog) serviceops (Incoming 🐫)	T292382: Add support for xhgui for jobs
T294024	T294024: [Airflow] Automate sync'ing archiva packages to HDFS	open	High (red)	Airflow (Done) Data-Engineering-Kanban (Ready to Deploy) Data-Engineering (Transform)	T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors	T282033: Airflow collaborations
T294026	T294026: [Airflow] Create repository for Airflow DAGs	open	High (red)	Data-Engineering-Kanban (Done) Airflow (Done) Data-Engineering (Transform)		T282033: Airflow collaborations
T297460	T297460: Send cassandra3 (new hosts) logs to logstash	open	High (red)	Data-Engineering-Kanban (Done) Data-Engineering (Ops)		T249755: Cassandra3 migration for Analytics AQS
T297803	T297803: Switch over the Cassandra AQS cluster to the new hosts	open	High (red)	Platform Team Workboards (Platform Engineering Reliability) (Backlog) Analytics (Analytics-Clusters) (Backlog) Epic Cassandra (Backlog) Data-Engineering (Serve) Data-Engineering-Kanban (Paused)		T249755: Cassandra3 migration for Analytics AQS
T298516	T298516: Investigate high levels of garbage collection on new AQS nodes	open	High (red)	Data-Engineering-Kanban (In Progress) Analytics (Analytics-Clusters) (Backlog) Cassandra (Backlog) Epic Data-Engineering (Serve) Platform Team Workboards (Platform Engineering Reliability) (Backlog)		T249755: Cassandra3 migration for Analytics AQS
T298805	T298805: Import Debian package of Cassandra 3.11.11 as 'dev' version	open	Needs Triage (violet)	Generated Data Platform (Backlog) SRE (Backlog) Data-Engineering (Serve) Platform Engineering (Inbox)