Difference between revisions of "T88991: improve Wikidata dumps [tracking]"

Latest revision as of 10:25, 2 September 2021

Phabricator Link	T88991
Status	open
Priority	High (red)
Points
Phabricator Task Created	2015/02/09 10:58 AM
Wiki Page Created	2019/06/09 01:29 PM
Phabricator Task Last Modified	2021/09/02 10:24 AM
Wiki Page Last Updated	2021/09/02 11:25 AM
Phabricator Task Closed
Authored By	Lydia_Pintscher (Lydia Pintscher)
Assigned To
Projects	Datasets-General-or-Unknown (Done) Tracking-Neverending (Tag) Wikidata (hold)
Platform Engineering Initiative Column
Platform Team Workboards Column
Subtasks	T211326: (Wikidata) Dump maintenance scripts should periodically reload DB configuration (open) T195419: Wikidata JSON dumps should not include Lexemes (resolved) T191639: Wikidata JSON dumps do not have the 'ns' (namespace) (invalid) T182348: dcatap.rdf in dumps contains invalid data (resolved) T177533: Consider sorting nt dumps and omit duplicate rows (declined) T155103: Create a truthy nt dump (resolved) T151876: Consider using pigz (Zopfli) for Wikidata JSON dump (open) T118379: bz2 dumps cannot be read with PHP (resolved) T89273: Produce stub dumps for all wikis as soon as a new month starts, then generate all other dumps on second round-robin cycle (resolved) T190513: Make sure Wikidata entity dump scripts run for only about 1-2hours (resolved) T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 (resolved) T177550: Allow dumping only entities in a certain page_id range in each maintenance script run (days: 1) (resolved) T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently (resolved) T119613: consider a serialization that supports random access for the dump (duplicate) T93658: cron job to create RDF dumps (resolved) T72385: Wikidata JSON dump: file directory location should follow standard patterns (resolved) T94019: Generate RDF from JSON (stalled) T162351: Create a "page prop" RDF dump for Wikidata entities ("pagePropertiesRdf") (open) T98320: [Task] Create dump of entity redirects (JSON or n-triples) (open) T87328: link to json dump documentation from dump site (resolved) T85407: improve dump documentation (resolved) T74348: Wikidata dumps contain old-style serialization. (invalid) T73349: [Bug] Fix empty map serialization behaviour (resolved) T73992: [Story] JSON should (optionally) contain full URIs for referenced external entities (open) T187888: "Failed to dump Q12129 (Value must be at most 127 characters long.)" when dumping Wikidata as TTL (resolved) T185589: Repeating blank node ids in Wikidata entity RDF dumps (resolved) T178978: Include information about Wikidata dumps in Wikidata query service (resolved) T176844: Look into truthy nt dump performance (resolved) T144103: Create .nt (NTriples) dumps for wikidata data (resolved) T91117: [Bug] Empty JSON maps are serialized as empty lists in XML dumps (resolved) T87329: update and consolidate json dump documentation (resolved) T87283: Wikidata dumps should have revision ID or other sequence mark (open) T85970: Some Wikidata XML dumps are failing (resolved) T85133: Introduce setting to allow import of entities from XML dumps. (resolved) T72247: provide "latest" link for JSON dumps (resolved) T70792: Wikidata JSON dump: filename prefix (resolved) T70366: add script for stitching json dump shards (resolved) T56085: [Task] EntityIdValues should be serialized as strings, not type/number structures. (resolved) T46580: Script for creating RDF dumps of all entities (resolved) T179681: Add HDT dump of Wikidata (stalled) T115223: Provide wikidata downloads as multiple files to make access more robust and efficient (open) T85101: create index for each dump (open) T72246: provide incremental JSON dumps for Wikidata (open) T46581: Partial dumps (open) T119612: Consider a serialization that supports random access for storage in the DB for Wikidata (open)
Parent Tasks	T88728: Improve Wikimedia dumping infrastructure

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Wikidata	⇐ incoming	9 February 2015
Wikidata	⇒ hold	9 February 2015
Datasets-General-or-Unknown	⇐ Backlog	2 July 2018
Datasets-General-or-Unknown	⇒ Done	2 July 2018

@@ Line 3: / Line 3: @@
 |status=open
 |color=red
+|points=
 |dateCreated=1423497524
-|dateModified=1559087517
+|dateModified=1630592679
 |author={{Phabricator User
 |type=author
@@ Line 13: / Line 14: @@
 |name=Datasets-General-or-Unknown
 |column=Done
+|entryDate=1530535146
 }}{{Phabricator Project
 |name=Tracking-Neverending
 |column=Tag
+|entryDate=1423497524
 }}{{Phabricator Project
 |name=Wikidata
 |column=hold
+|entryDate=1423510902
+}}
+|subtasks=290244,241169,229290,211326,195419,191639,182348,177533,155103,151876,118379,89273,190513,181385,177550,177486,119613,93658,87328,85407,74348,73349,72385,187888,185589,178978,176844,162351,144103,98320,94019,91117,87329,87283,85970,85133,73992,72247,70792,70366,56085,46581,46580,179681,115223,85101,72246,119612
+|transitions={{Transition
+|date=1530535146
+|type=Entered Column
+|project=Datasets-General-or-Unknown
+|column=Done
+}}{{Transition
+|date=1530535146
+|type=Exited Column
+|project=Datasets-General-or-Unknown
+|column=Backlog
+}}{{Transition
+|date=1423510902
+|type=Entered Column
+|project=Wikidata
+|column=hold
+}}{{Transition
+|date=1423510902
+|type=Exited Column
+|project=Wikidata
+|column=incoming
 }}
-|subtasks=211326,195419,191639,182348,177533,155103,151876,118379,89273,190513,181385,177550,177486,119613,93658,87328,85407,74348,73349,72385,187888,185589,178978,176844,162351,144103,98320,94019,91117,87329,87283,85970,85133,73992,72247,70792,70366,56085,46580,179681,115223,85101,72246,46581,119612
 }}