← Benchmarks | Materials Physics

Materials Benchmark CORE + UNIVERSAL

Two benchmark tracks: core 5-property holdout splits (S2/S3, with the evaluation configuration declared) and the Universal 16 in-sample consistency suite, with inputs and calibration provenance declared. This page also includes a curated mini-benchmark for defect-context gemstone color behavior.

1.1668%
Family Holdout Error (S2)
Held out by family; engine not refit per fold
1.3750%
Interaction Holdout Error (S3)
Held out by interaction type; engine not refit per fold
0.32-0.33 ms
Core S2/S3 Runtime Mean
Per prediction call
797 / 821
Scored Property Points
Total points in S2 / S3

Core Holdout Results (5 Properties)

Deterministic production runtime. No ML inference in final calculations.

Scope. Materials are held out by family (S2) or interaction type (S3), but the engine is not refit per fold: calibration terms set on held-out materials stay active. These figures are therefore not a clean measure of accuracy on new materials. Accuracy on new materials is reported in the dedicated property benchmarks.
Test Overall MAPE Bulk Modulus (B) Debye Temp (theta_D) Sound Velocity Density Thermal Cond. (kappa) Runtime mean
Test A: Family holdout (S2) 1.1668% 0.9056% 1.3283% 1.4405% 1.2231% 0.9185% 0.3294 ms
Test B: Interaction holdout (S3) 1.3750% 1.0149% 1.5636% 1.6081% 1.3398% 1.3473% 0.3205 ms

Units: bulk modulus (GPa), Debye temperature (K), sound velocity (m/s), density (g/cm^3), thermal conductivity (W/(m*K)).

Universal 16: In-Sample Consistency Suite

February 2026 suite: inputs, reference overlap and evaluation protocols

What this suite measures. The complete February 2026 engine reproduces its reference set with all published strict rows below 1%. Calibration for some properties used these same reference rows. This result measures agreement with the declared reference cohort. Accuracy on new cases is assessed separately through property-specific benchmarks and independent evaluation.
<1%
Strict MAPE, as deployed
in-sample, all 16 rows
14
Distinct quantities
N from 28 to 274 per property
856
Unary and binary compounds
743 with a prediction
17 / 17
Rows reproduced exactly
evaluation 23 Oct 2026
Property Strict MAPE % (N), as deployed Out-of-family weighted MAPE % (rows)
Crystal structure0.000 (59)0.000 (59)
Lattice constant (A)0.085 (59)0.086 (59)
Lattice constant (pm)0.085 (59)0.086 (59)
Atomic volume (A^3)0.497 (71)0.488 (71)
Band gap (eV)0.690 (274)0.985 (674)
Optical gap (eV)0.527 (39)0.530 (39)
Dielectric constant0.910 (32)0.925 (32)
Refractive index0.658 (49)0.662 (49)
Reflectivity0.339 (28)0.338 (28)
Hardness (GPa)0.740 (44)0.724 (44)
Cv (J/mol-K)0.857 (39)0.858 (39)
Cv (J/kg-K)0.857 (39)0.858 (39)
Cp (J/mol-K)0.797 (81)0.801 (81)
CTE (1/K)0.903 (39)0.908 (39)
Melting point (K)0.947 (82)0.973 (82)
Density (g/cm^3)0.929 (162)0.965 (165)

16 published rows cover 14 distinct quantities: lattice constant and Cv each appear in two units. The 17-row reproduction also expresses thermal expansion in ppm/K. Crystal structure: 59 of 59 classes correct (categorical). The strict band-gap reference includes 201 measured and 73 calculated values (274 total). These suite results measure in-sample agreement in the declared configuration; property benchmarks assess their own inputs and cohorts.

Inputs

Chemical formula, plus the material category and cohesive energy read from the same reference row that holds the experimental value. The production interface receives the formula, derives the category and can resolve packaged cohesive-energy context for known materials. Formula entry alone does not imply that the calculation is independent of reference data. Each property benchmark declares the inputs and reference contributions of its complete system.

Strict

A frozen list of unary and binary formulas scored against the reference rows, engine as deployed. No split and no holdout. Rows without a prediction and zero references are excluded from the MAPE.

Out-of-family

15 ways of partitioning the same materials (category, anion family, block, period, crystal system, coordination, bonding regime and others), leave-one-group-out, weighted MAPE. The engine keeps all calibration terms, including those for held-out materials, and the population adds rows with extra data (band gap: 674 vs 274).

Mini-Benchmark: Defect-Context Gemstone Color

A curated showcase of defect-center and illumination behavior using the documented deterministic runtime path.

How to interpret this: this is a nice-to-have flexibility benchmark for defect-driven visible color. It is a curated scenario set, not a replacement for the Universal reference-cohort consistency suite above.
19 / 19
Gemstone Color Match
Exact or accepted family label
2.800 ms
Median Runtime
Per prediction call
Green to Red
Alexandrite Shift
Daylight vs incandescent
21
Knob Sweep Cases
5 output color classes observed
Case Context Expected Predicted
Amethyst Quartz + Fe center Purple Purple
Ruby Corundum + Cr center Red Red
Emerald Beryl + Cr/V center Green Green
Blue Topaz Topaz + F-like center Blue Blue
Alexandrite (daylight) Cr center, daylight profile Green Green
Alexandrite (incandescent) Same material, warm-light profile Red Red

Runtime remains deterministic and constant-time. No ML inference is used in final engine calculations.

How to read this page

What is measured

Core track: 5 thermo-mechanical properties. Universal track: 16 published rows covering 14 distinct quantities across structural, electronic, optical, thermal, and mechanical outputs.

How accuracy is scored

MAPE (%). Lower is better.

How speed is scored

Mean runtime in milliseconds per prediction call.

Mini-benchmark scope

The gemstone color section is a curated feature benchmark that demonstrates defect and illumination controls. It is separate from core strict property claims.

Two Holdout Splits

Materials are held out by family or by interaction type; the engine is not refit per fold.

Test A: Family Holdout (S2)

All materials from one family are held out together. Calibration terms set on held-out materials stay active.

  • 19 folds
  • 171 held-out formulas total
  • 797 total evaluated property points

Test B: Interaction Holdout (S3)

Bonding interaction types are held out by group. The February evaluation keeps the deployed calibration active, including terms associated with evaluated materials.

  • 15 folds
  • 175 held-out formulas total
  • 821 total evaluated property points

Reproducibility and Audit Trail

Both benchmark tracks are frozen to public snapshots so every number can be rerun and verified.

Snapshot ID: materials_physics_benchmark_snapshot_2026-02-24
Generated: 2026-02-24T20:50:56Z
Commit: f4fb848fd7fa55be1b68d4e7592f1330553f1112
Branch / Dirty: main / true

Dirty: true indicates public report-formatting/export changes after the prediction rows were frozen. The benchmark prediction rows, snapshot ID, commit, commands, and attached hashes define the reproducible artifact.

Frozen benchmark JSON
Machine-readable manifest, hashes, commands, and metrics.
Download JSON
Frozen benchmark report (Markdown)
Human-readable report with strict and external tables.
Download MD
Universal benchmark snapshot JSON (16 rows, 14 quantities)
Strict + out-of-family aggregates with sanitized public fields only.
Download JSON
Universal benchmark snapshot report (Markdown)
Headline metrics and per-property strict vs out-of-family summary table.
Download MD
Strict scoring outputs (S2/S3)
Fold-level strict results for FLUX only.
S2 JSON S3 JSON
External apples-to-apples outputs (S2/S3)
FLUX + AFLOW/JARVIS/Matbench adapter scoring on the same strict splits.
S2 JSON S3 JSON

Dedicated materials benchmarks

Specific property accuracy and methodology details are reported on dedicated pages with their own datasets and metric definitions.

Band Gap Benchmark Crystal Bond Lengths Curie Temperature Materials Module

Benchmark basis

This page aggregates many materials properties. The aggregate basis is mixed; property rows and source notes identify the relevant prediction route for each result family.

Mixed basis