Skip to main content

Slovenian Lexicographic Datasets

Slovenian Lexicographic Datasets — dict-conversions

Two LM-ready datasets built from 15 legacy CJVT/DZS Slovenian dictionaries (14 bilingual + 1 monolingual encyclopedia), converted to OASIS DMLex 1.0 and HuggingFace Parquet, then enriched. This document describes both datasets, the pipeline that produced them, their schemas, and what to look for when using them.

Status: 805,279 entries · 1,019,685 senses · 2,174,880 translation pairs · 113,868 relations. All 15 dictionaries validate against the official DMLex XSD 1.1 + JSON Schema with 0 leaks, 0 parse errors, 0 residual glyph markers. 52 unit tests pass.

1. The two datasets at a glance

Collection 1 — Core Collection 2 — Enriched
Path dist/core/ (+ canonical lexidma/, parquet/) dist/enriched/ (extends Core)
Provenance Intrinsic — derived only from the project's own dictionaries Extrinsic — Core + external resources
External tools none CLASSLA-Stanza, sloWNet/OMW, English WordNet (oewn)
Size ~2.1 GB ~184 MB (layers only; use with Core)
Reproducible offline yes needs the external resources (one-time download)
Contents DMLex XML/JSON, derived tables, 12 LM task JSONLs silver morphology + MSD task, synset/ILI links, imported antonyms, candidate scoring, sloWNet-enriched KNAUR DMLex

Use them together. Enriched is a thin layer of external-resource columns/files that extends Core; it does not duplicate it.

2. Source corpus

sh = Serbo-Croatian (legacy unified tag). KNAUR is monolingual (definitions + cross-references, no translations). DVRUSL is reconstructed from a Word .doc (OCR-grade).

Code Languages Family Entries Senses Pairs Relations
DVANSL en→sl block_blankline 68,761 75,208 264,454 0
DVFRSL fr→sl line 39,980 70,817 175,458 0
DVITSL it→sl block_blankline 60,282 98,444 197,449 0
DVRUSL ru→sl doc_runs 32,844 32,844 80,364 0
DVSHSL sh→sl block_blankline 92,302 92,302 150,228 5,563
DVSPSL es→sl dzs_utf16 36,460 66,722 114,691 414
DVSLAN sl→en block_blankline 42,088 42,227 159,625 0
DVSLFR sl→fr line 33,866 44,092 128,561 34
DVSLNE sl→de german 84,951 90,374 252,074 120
DVSLSH sl→sh block_blankline 72,690 87,140 152,068 2,694
DVSLSP sl→es dzs_utf16 33,676 66,456 93,562 349
DRSLAN sl→en dzs_nested 25,559 33,273 75,374 457
VSIS sl→it block_geslo 90,675 116,223 275,043 7,764
LAT_AZ la→sl block_geslo 11,521 20,315 55,929 1,852
KNAUR sl (mono) encyclopedia 79,624 83,248 0 94,621

Several pairs exist in both directions (DVSLFR/DVFRSL, DVSLAN/DVANSL, DVSLSH/DVSHSL, DVSLSP/DVSPSL, VSIS+DVITSL) — exploited for translation-pivot synonyms and reverse-dictionary tasks.

3. The pipeline

                 ┌───────── BASE CONVERSION (intrinsic, faithful) ──────────┐
 source dicts ─► parser family ─► IR (model.py) ─► DMLex XML/JSON  (lexidma/)
   (8 families)   + textproc                     └► Parquet         (parquet/)  + reports/
                  (specialchars, markup,
                   controlled vocab)
                          │
                          ▼
      ┌──────── COLLECTION 1 · CORE (build/, intrinsic) ────────┐
      │ derive.py  : accent-folded lemma · dedup · leak-free split
      │ synsets.py : in-sense synonyms · translation pivots · KNAUR hypernyms · GWA relation typing
      │ tasks.py   : 12 LM task JSONLs                                   → dist/core/
      └─────────────────────────────────────────────────────────┘
                          │
                          ▼
      ┌──────── COLLECTION 2 · ENRICHED (build/enrich.py, extrinsic) ───────┐
      │ CLASSLA silver lemma/UPOS/MSD (+ msd_tagging task)
      │ sloWNet/OMW synset+ILI links · ILI-bridged antonyms
      │ candidate scoring vs sloWNet · sloWNet-typed DMLex relations (KNAUR) → dist/enriched/
      └─────────────────────────────────────────────────────────────────────┘

Stage 1 — Base conversion (src/dictconv/, command dictconv convert all). Each dictionary's publisher typesetting markup is tokenized (not XML-parsed — it is malformed), {…} escapes decoded to Unicode (side-aware accents; undecodable font/template codes removed and logged), qualifiers classified into a controlled vocabulary, and emitted as a source-agnostic intermediate representation. Serializers produce DMLex 1.0 XML+JSON (validated against the official OASIS XSD 1.1 / JSON Schema) and three Parquet artifacts. The conversion is faithful: it never invents content and flags rather than silently drops.

Stage 2 — Core (dictconv build-core). Adds derived, ML-oriented layers computable from our data alone: a normalized lemma/UPOS layer, exact/near-duplicate collapse, a leak-free train/dev/test split, synonym/pivot/hypernym candidate tables, and 12 instruction-style task JSONLs.

Stage 3 — Enriched (dictconv enrich). Adds layers that need outside resources, kept strictly separate from the intrinsic data (gold-vs-silver provenance is explicit).

4. Base conversion artifacts

lexidma/<CODE>.xml and .json — OASIS DMLex 1.0

Faithful lexicographic record. Camel-case elements; text of every object except headword in a nested <text>; crosslingual module (headwordTranslation, exampleTranslation); the linking module (relation + relationType definitions that carry a Global WordNet sameAs URI, e.g. see → https://globalwordnet.github.io/schemas/wn#also). Validated with a real XSD 1.1 processor (all identity constraints + assertions, except 3 cardinality-defective ones the published schema cannot satisfy). KNAUR uses the monolingual schema.

parquet/<CODE>.entries.parquet — one row per entry (nested)

dict_code, entry_id, headword,
lemma,            # accent-folded join key (Sloleks/Gigafida/CLASSLA/sloWNet keying)
accented_form,    # original tonal/accented display form (null if == headword)
homograph_number, source_lang, target_lang,
meta_lang,        # editorial metalanguage = "sl"
parts_of_speech[str], upos,    # UPOS from the entry's first POS (intrinsic static map)
frequency_band,   # DRSLAN corpus band 0..3 (null elsewhere)
labels[str], collocates[str],  # DRSLAN <KO> collocates
pronunciations[{text, scheme}],
inflected_forms[{text, tag}],
senses[{ sense_id, indicator, labels[str], definitions[str],
         headword_translations[{text, lang_code, parts_of_speech[str], labels[str]}],
         headword_explanations[{text, lang_code}],
         examples[{text, labels[str], translations[{text, lang_code, labels[str]}]}] }],
has_content,      # False => no senses / all senses empty (filter before LM use)
source_ref, raw   # provenance

parquet/<CODE>.pairs.parquet — one row per (source,target) unit (flat)

dict_code, source_lang, target_lang, entry_id, sense_id, homograph_number,
pair_type (headword|example), source_text, target_text,
source_lemma,    # accent-folded entry headword (dedup + leak-free split key)
part_of_speech, labels[str], domain, register

parquet/<CODE>.relations.parquet — the full cross-reference graph

dict_code, source_lang, target_lang, relation_index, type, description,
members[{ref, headword, role, target_id}],
serialized       # True => >=2 members resolved => present in DMLex XML/JSON

This is the lossless home of the cross-ref graph: it keeps cross-references whose target never resolved to an entry id (which the DMLex XML/JSON must drop).

reports/<CODE>.report.json + reports/_summary.json

Per-dictionary stats, validation results, the controlled-value inventory, flagged-token counts, and the aggregate summary + artifact manifest (sha256 of every output).

5. Collection 1 — Core (dist/core/)

5.1 Derived tables (dist/core/derived/)

File Rows What it is
lemmas.parquet 805,279 one row per entry: lemma, accented_form, upos, frequency_band, cluster_id, split
pairs_dedup.parquet 2,078,214 de-duplicated translation pairs + occurrence_count, canonical_id, split
synonym_sets.parquet 378,783 in-sense (target-language) near-synonym sets + gloss
synonym_pairs.parquet 2,131,996 in-sense + pivot synonym pairs (evidence, confidence_tier)
pivot_synonyms.parquet 152,655 Slovene synonym candidates from translation pivots (GOLD 67,296 / SILVER 85,359)
hypernym_candidates.parquet 4,169 KNAUR genus-differentia hypernym candidates (confidence)
relations_typed.parquet 113,868 the cross-ref graph, GWA-typed (gwa_relType)

Pivot-synonym yield: 152,655 SILVER+GOLD pairs (≥2 agreeing pivots) materialized; 632,231 single-pivot BRONZE pairs were counted but not materialized (large, low precision); 339,415 distinct pivots used.

5.2 LM tasks (dist/core/tasks/*.jsonl)

Each row: {id, task, split, input:{…}, output:{…}, metadata:{…}}. Split is train/dev/test (≈90/5/5), leak-free (§5.3). Marker policy drop (undecodable-glyph rows are cleaned/omitted).

Task Rows input → output
translation 3,006,506 {source_text, source_lang, target_lang, part_of_speech, labels, domain, register} → {target_text} (both directions)
example_translation 574,961 example phrase → its translation
definition 145,633 {headword, lang, indicator} → {definition} (KNAUR)
reverse_dictionary 145,633 {definition, lang} → {headword}
wsd 22,036 {word, context, lang} → {sense_gloss, sense_id} (polysemous only; bare-number glosses dropped)
example_usage 85,155 {headword, lang} → {example} (monolingual usage sentences)
morphology 332,393 {headword, lang} → {form, tag} (dictionary inflected forms)
pronunciation 157,578 {headword, lang} → {transcription, scheme}
synonyms_of 358,573 {word, lang} → {synonyms[]} (a real set; 60% have >1)
hypernym_of 4,169 {word, lang} → {hypernym_candidate, confidence}
relation 113,373 a relation's first member → {relation_type, members[…]}
relation_classify 113,327 {a, b, lang} → {relation_type} (unordered-pair split)

5.3 Leak-free split (important)

  • Translation / sense / synonym tasks key on the folded Slovene lemma, so a lemma and its reverse-direction twin (hiša in sl→fr and maison→hiša in fr→sl) are always in the same split. Verified: 0 of 172,081 Slovene headword lemmas straddle splits. (e.g. translation split: train 2,711,054 / dev 144,596 / test 150,856.)
  • Morphology & pronunciation (about the headword form) split by headword form; relation_classify by the unordered member pair — so foreign homographs don't straddle either.
  • The legacy dict_code:entry_id key (now superseded) leaked ~26 % of multi-dict lemmas.

5.4 Cleaning applied (Core)

Dedup before split (with occurrence_count); degenerate targets dropped (punct/digit-only, single-char, src==tgt); unbalanced parentheses balanced; PUA sentinels + control chars stripped; undecodable glyph markers removed (marker_policy=drop). manifest.json carries a content hash (e76f2766…) and per-task split counts; dataset_card.md is the in-tree card.

6. Collection 2 — Enriched (dist/enriched/)

File Rows What it is
silver_morphology.parquet 208,715 CLASSLA lemma / UPOS / JOS-MULTEXT-East MSD / feats per Slovene lemma; morph_provenance="silver_tool"
tasks/msd_tagging.jsonl 208,715 {lemma, lang} → {upos, msd, feats} (the morphology/POS task; silver)
synset_links.parquet 64,413 Slovene lemma → sloWNet/OMW synset_id + ILI (join key to Princeton WN / OMW)
antonyms.parquet 6,107 imported Slovene antonyms (ILI-bridged through the English WordNet)
scored_synonyms.parquet 364,334 every Core synonym candidate + wordnet_confirmed + source_count
scored_hypernyms.parquet 396 checkable hypernym candidates + wordnet_confirmed
lexidma/KNAUR.{xml,json} 97,418 rel KNAUR re-serialized as DMLex with sloWNet antonym (142) + synonym (2,655) relations, ILI/synset in relation/description, GWA-typed

6.1 Candidate scoring vs sloWNet (measured precision — lower bounds; sloWNet is incomplete)

  • Synonyms: 364,334 checkable, 17.0 % confirmed — in-sense 14.6 %, pivot 27.1 %, pivot-GOLD 38.5 %.
  • Hypernyms: 396 checkable, 47.5 % confirmed (ILI-bridged through the English WordNet).
  • Use wordnet_confirmed=True (and/or confidence_tier=GOLD) to extract a higher-precision subset.

6.2 External resources & how to reproduce

sloWNet/antonyms run in the main .venv (pip install -e '.[enrich]' → wn). CLASSLA needs Python ≤ 3.13 (its pinned numpy fails to build on 3.14), so run the silver morphology from a 3.12 env:

uv venv --python 3.12 .venv-enrich
uv pip install -p .venv-enrich/bin/python -e '.[enrich]'
python -m wn download omw-sl ; python -m wn download oewn:2021
.venv-enrich/bin/python -c "import classla; classla.download('sl')"
.venv-enrich/bin/python -m dictconv.cli enrich --in dist/core --out dist/enriched --sample-limit 0

7. What to look for (usage guidance & caveats)

Filter before training

  • has_content — drop entries with no usable content (≈1.2 % of entries) for entry-level tasks.
  • marker_policy — task JSONLs are already built with drop; never train on the keep variant (it would teach the model to emit placeholder glyphs). Corpus markers are currently 0.
  • Degenerate rows — already removed from the tasks; if you build your own from parquet/, apply the same filters (punct/digit-only, src==tgt, unbalanced parens).
  • Use the provided split. Re-shuffling by row re-introduces lemma leakage; the cluster split is the point. Hold out whole lemma clusters, not rows.

Candidates are candidates, not gold

  • synonym_*, pivot_synonyms, hypernym_candidates are induced and noisy. Gate with the enriched scoring: scored_synonyms.wordnet_confirmed / pivot confidence_tier=GOLD (38.5 % precision) for synonyms; scored_hypernyms.wordnet_confirmed for hypernyms. The 47.5 % hypernym figure is measured on a small checkable slice and is optimistic for the full pool (genus heads are not lemmatized — ~25 % are oblique forms; lemmatize with CLASSLA before use).
  • Antonyms are imported, not mined (synonyms/antonyms are translationally indistinguishable).
  • Precision is measured only on vocabulary sloWNet already has (~11 %). The extension value (the ~89 % of members not yet in sloWNet) is unproven — commission a small human eval before treating those as silver.

Gold vs silver

  • The gold lemma layer (lemmas.parquet) is 100 % coverage, accent-folded, NFC-clean.
  • The silver morphology (silver_morphology / msd_tagging) is CLASSLA tool output (morph_provenance="silver_tool"). Keep it filterable; measure MSD accuracy on a hand-tagged sample before training a morphological analyzer on it. The dictionary's own inflected_forms are mostly ending fragments, not full words.

Per-dictionary quality

  • DVRUSL (Russian) is OCR-grade (Word .doc reconstruction). Cleanups were applied (brace→paren, |/bullet stripping, space collapse) but residual noise is inherent — down-weight or exclude for high-precision work.
  • Definitions / reverse-dictionary come only from KNAUR (monolingual Slovene). There are no definitions from the 14 bilingual dicts.
  • Conditioning labels are sparse (POS on ~18 % of translation rows, domain ~4 %, register ~4 %).

Provenance / audit

  • Every removed escape token is logged in data/reference/removed_markers.tsv.
  • reports/_summary.json carries the per-file sha256 manifest; each collection's manifest.json carries a content_hash and split counts. Pin these with any eval run.

8. Loading

from datasets import load_dataset

# flat translation pairs (all dicts)
pairs = load_dataset("parquet", data_files="parquet/*.pairs.parquet", split="train")

# nested per-entry records (one dictionary)
entries = load_dataset("parquet", data_files="parquet/VSIS.entries.parquet", split="train")
entries = entries.filter(lambda r: r["has_content"])          # drop empty entries

# a Core LM task, by split
import json
train = [json.loads(l) for l in open("dist/core/tasks/translation.jsonl") if json.loads(l)["split"]=="train"]

# wordnet-confirmed synonyms only (Enriched gate)
import pyarrow.parquet as pq
syn = pq.read_table("dist/enriched/scored_synonyms.parquet").to_pylist()
gold = [r for r in syn if r["wordnet_confirmed"]]

# DMLex (faithful lexicographic view)
import json; res = json.load(open("lexidma/VSIS.json"))       # OASIS DMLex 1.0

9. Reproduce end-to-end

pip install -e .                 # base deps (pyarrow, lxml, jsonschema, xmlschema)
dictconv convert all --write-summary     # base: lexidma/, parquet/, reports/
dictconv build-core                       # Collection 1 -> dist/core/
dictconv enrich --sample-limit 0          # Collection 2 -> dist/enriched/  (see §6.2 for CLASSLA env)
dictconv audit --write-summary            # readiness audit + artifact manifest
pytest -q                                 # 52 tests
Back to top