T88991: improve Wikidata dumps [tracking]

Phabricator Link	T88991
Status	open
Priority	High (red)
Points
Phabricator Task Created	2015/02/09 10:58 AM
Wiki Page Created	2019/06/09 01:29 PM
Phabricator Task Last Modified	2021/09/02 10:24 AM
Wiki Page Last Updated	2021/09/02 11:25 AM
Phabricator Task Closed
Authored By	Lydia_Pintscher (Lydia Pintscher)
Assigned To
Projects	Datasets-General-or-Unknown (Done) Tracking-Neverending (Tag) Wikidata (hold)
Platform Engineering Initiative Column
Platform Team Workboards Column
Subtasks	T211326: (Wikidata) Dump maintenance scripts should periodically reload DB configuration (open) T195419: Wikidata JSON dumps should not include Lexemes (resolved) T191639: Wikidata JSON dumps do not have the 'ns' (namespace) (invalid) T182348: dcatap.rdf in dumps contains invalid data (resolved) T177533: Consider sorting nt dumps and omit duplicate rows (declined) T155103: Create a truthy nt dump (resolved) T151876: Consider using pigz (Zopfli) for Wikidata JSON dump (open) T118379: bz2 dumps cannot be read with PHP (resolved) T89273: Produce stub dumps for all wikis as soon as a new month starts, then generate all other dumps on second round-robin cycle (resolved) T190513: Make sure Wikidata entity dump scripts run for only about 1-2hours (resolved) T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007 (resolved) T177550: Allow dumping only entities in a certain page_id range in each maintenance script run (days: 1) (resolved) T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently (resolved) T119613: consider a serialization that supports random access for the dump (duplicate) T93658: cron job to create RDF dumps (resolved) T72385: Wikidata JSON dump: file directory location should follow standard patterns (resolved) T94019: Generate RDF from JSON (stalled) T162351: Create a "page prop" RDF dump for Wikidata entities ("pagePropertiesRdf") (open) T98320: [Task] Create dump of entity redirects (JSON or n-triples) (open) T87328: link to json dump documentation from dump site (resolved) T85407: improve dump documentation (resolved) T74348: Wikidata dumps contain old-style serialization. (invalid) T73349: [Bug] Fix empty map serialization behaviour (resolved) T73992: [Story] JSON should (optionally) contain full URIs for referenced external entities (open) T187888: "Failed to dump Q12129 (Value must be at most 127 characters long.)" when dumping Wikidata as TTL (resolved) T185589: Repeating blank node ids in Wikidata entity RDF dumps (resolved) T178978: Include information about Wikidata dumps in Wikidata query service (resolved) T176844: Look into truthy nt dump performance (resolved) T144103: Create .nt (NTriples) dumps for wikidata data (resolved) T91117: [Bug] Empty JSON maps are serialized as empty lists in XML dumps (resolved) T87329: update and consolidate json dump documentation (resolved) T87283: Wikidata dumps should have revision ID or other sequence mark (open) T85970: Some Wikidata XML dumps are failing (resolved) T85133: Introduce setting to allow import of entities from XML dumps. (resolved) T72247: provide "latest" link for JSON dumps (resolved) T70792: Wikidata JSON dump: filename prefix (resolved) T70366: add script for stitching json dump shards (resolved) T56085: [Task] EntityIdValues should be serialized as strings, not type/number structures. (resolved) T46580: Script for creating RDF dumps of all entities (resolved) T179681: Add HDT dump of Wikidata (stalled) T115223: Provide wikidata downloads as multiple files to make access more robust and efficient (open) T85101: create index for each dump (open) T72246: provide incremental JSON dumps for Wikidata (open) T46581: Partial dumps (open) T119612: Consider a serialization that supports random access for storage in the DB for Wikidata (open)
Parent Tasks	T88728: Improve Wikimedia dumping infrastructure

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Wikidata	⇐ incoming	9 February 2015
Wikidata	⇒ hold	9 February 2015
Datasets-General-or-Unknown	⇐ Backlog	2 July 2018
Datasets-General-or-Unknown	⇒ Done	2 July 2018