# Data Dictionary

This dictionary describes the core public dataset and the larger standardized observation layer used in the processing workspace. Public downloads preserve source values, units and Chinese labels.

## Core indicators: `key_indicators_latest.parquet`

The public CSV, JSON and Parquet core files use the following fields.

| Field | Type | Meaning |
|---|---|---|
| `metric_id` | string | Stable website/API indicator key. |
| `metric_name_zh` | string | Chinese indicator name. |
| `category` | string | `capacity`, `generation` or `consumption`. |
| `energy_type` | string | `total`, `hydro`, `thermal`, `nuclear`, `wind` or `solar`. |
| `measure` | string | `value` or `share`. |
| `region_name` | string | National or provincial region name in Chinese. |
| `region_code` | string | `CHN` nationally; six-digit provincial code. |
| `year` | int64 | Statistical year. |
| `value` | double | Numeric value reported in the original yearbook. |
| `unit` | string | Source unit: 10,000 kW, 100 million kWh, or %. |
| `edition_year` | int64 | Latest available yearbook edition used for this value. |
| `source_relative_path` | string | Relative path to the source PDF. |
| `source_page` | int64 | Source PDF page number. |
| `table_title` | string | Source table title. |
| `table_id` | string | Source table identifier. |
| `quality_status` | string | Automatic extraction from a structured table; source sampling remains advisable. |

### Unit conversions

| Original label | Meaning | Conversion |
|---|---|---|
| `万千瓦` | 10,000 kW | Multiply by 10,000 for kW, or by 0.01 for GW. |
| `亿千瓦时` | 100 million kWh | Multiply by 100,000,000 for kWh, or by 0.1 for TWh. |
| `%` | Percentage points | Retain as percentage values. |

Perform conversions in a new derived field and retain the original `value` and `unit`. The English website labels these units explicitly, so the displayed numbers retain the same interpretation.

## Standardized observations: `observations_standardized.parquet`

This larger processing layer combines Excel and PDF observations. It is distinct from the lightweight core download package.

| Field | Type | Meaning |
|---|---|---|
| `observation_id` | string | Stable observation identifier. |
| `source_type` | string | `excel` or `pdf_text`. |
| `collection` | string | Chinese energy or electricity yearbook collection. |
| `edition_year` | int16 | Publication edition; may differ from the statistical year. |
| `source_id` | string | Source-file identifier. |
| `source_relative_path` | string | Source path relative to the original data archive. |
| `table_id` | string | Table or worksheet identifier, joined to the table catalogue. |
| `sheet_name` | string | Excel worksheet name; empty for PDF records. |
| `page` | int16 | PDF page number; empty for Excel records. |
| `table_title` | string | Automatically identified table title. |
| `metric_auto` | string | Automatically composed indicator label; review against the indicator dimension before formal use. |
| `row_label` | string | Original row label. |
| `column_label` | string | Original column label. |
| `region_name` | string | Standardized region name; empty if identification is uncertain. |
| `region_code` | string | National or provincial code; national records use `CHN`. |
| `data_year` | int16 | Statistical year; empty if parsing is uncertain. |
| `value` | double | Numeric value, without implicit unit conversion. |
| `unit_raw` | string | Original detected unit. |
| `unit_normalized` | string | Standardized unit label. |
| `unit_dimension` | string | Physical dimension of the unit. |
| `quality_tier` | string | Automatic quality category. |
| `quality_flags` | string | Extraction flags separated by `|`. |
| `source_row` | int32 | Original Excel row number. |
| `source_column` | int16 | Original Excel column number. |
| `x0` | float | Left coordinate of the PDF word/value box. |
| `y0` | float | Top coordinate of the PDF box. |
| `x1` | float | Right coordinate of the PDF box. |
| `y1` | float | Bottom coordinate of the PDF box. |

## Keys and relationships

- Standardized-observation primary key: `observation_id`.
- Table-catalogue primary key: `table_id`, also present in observations.
- Core natural key: `metric_id + region_code + year`; verified to contain no duplicates.
- `edition_year` is a publication year; `data_year` and `year` are statistical periods.
- Use the unit dimension's `multiplier_to_base` for conversions without overwriting original values.

## Quality categories

- `auto_extracted`: year and unit parsed automatically; not a claim of manual verification.
- `needs_year_review`: year could not be parsed reliably; unsuitable for direct time-series use.
- `needs_unit_review`: missing or uncertain unit; unsuitable for direct cross-table comparisons.
- `auto_extracted_structured_table`: core indicator extracted from a structurally stable recent PDF table with source pages retained; source sampling remains recommended.