Reproducing Kobayashi et al. 2017: interface states on SiC after an argon anneal

One of the [studies we reproduced](/writing/interface-studies-reproduced) for [Questions we need Emily to answer](/writing/questions-for-emily-testing-a-theory-of-the-interface), the companion to [the essay](/writing/testing-a-theory-of-the-interface-with-an-ontology). Generated by `scripts/reproduce_studies.py` from `reproductions/kobayashi_2017.py`. Every value from the paper was read by an AI agent and has not been checked by a person; the sample and history were re-read from the PDF alone by a second agent that had not seen our earlier transcription.

The study and why we depend on it

T. Kobayashi, J. Suda and T. Kimoto, “Reduction of interface state density in SiC (0001) MOS structures by post-oxidation Ar annealing at high temperature”, AIP Advances 7, 045008 (2017).

The only electrical data we hold on SiC: 27 values of interface state density at three energies, taken from the vector drawing of its Figures 2 and 3. The SiC interface-state program (a one-way reduction set by the argon anneal temperature) was fitted to these and nothing else, so this page is a check on our reading, not a test of the model.

How well can this study be reproduced?

Low. The process is given as ranges for the whole sample set: the argon anneal is “1 - 10 min”, the 1300 °C oxidation “40 min or 1h 20 min”, the cooling rates are bounds, and what happens between the two furnaces is not described. Agreement here is with a program fitted to these same points.

The grade is ours and is about the paper as a source for simulation, not about the quality of the experiment: high means the sample, every step and its timing are stated well enough that a simulation needs no choices of ours; moderate that some choices are ours and are listed; low that the result depends on choices the paper does not constrain.

The sample and its history, played back

The sample is drawn from the simulation: layer heights and the read-outs are what the ISL programs compute at each moment of the history. Steps the language cannot state (a clean, a metallization, a measurement) are on the clock and marked as recorded, not simulated.

Does our sample and history match the paper's?

Left: the paper's own words, with the page of the PDF. Right: what we simulate. 4 same, 1 approximated, 1 assumed, 2 not in ours.

The timing of the history

Time is counted from the start of the first simulated step. A step the language cannot state takes no simulated time; where the paper gives its duration, the duration is in the last column. Stated means the paper gives the duration; partly stated that it gives a bound or a range; not stated; assumed that the duration in our history is ours.

Oxidized at 1150 °C, no anneal

Simulated time in this history: 20 h 00 min.

Across the samples

Timing the paper does not give

• The argon anneal time for each sample: only “1 - 10 min” for the whole set.
• Which 1300 °C oxidation time (40 min or 1 h 20 min) each sample had.
• Everything between the oxidation furnace and the anneal furnace: cooling, unloading, time in air, reloading and the ramp to 1300 or 1500 °C.
• The cooling rates are bounds (“< 70”, “> 600” °C/min), with no temperature interval and no gas. The two rates are also two different furnaces.
• Whether the twice-annealed sample was unloaded or measured between its two anneals.
• The temperature of the C-V measurement.

What the second reading changed

• Oxidation and argon anneal were done in different furnaces, so each annealed sample was cooled, unloaded and reheated in between. Our stored history runs the anneal straight after the oxidation.
• Five minutes of argon and forty minutes of oxidation at 1300 °C are our choices from the ranges the paper gives (1 to 10 min; 40 min or 1 h 20 min). The stored records present them as the recipe.
• Cooling rate is tied to furnace type: the slow rate is the resistive furnace (oxygen measured under 1 ppm), the fast rate the induction furnace (oxygen not reported). A difference between cooling rates is also a difference between furnaces.
• The substrate is 4° off-axis; the growth data we use for SiC (Song 2004) are on-axis. The sample records did not distinguish them.

Their terms and ours

The simulation in our language

Each sample is one history file: a SAMPLE and a CONTROL_TRAJECTORY. This is the whole input; run it with PYTHONPATH=. python -m isl_harness.predict <file>.

Oxidized at 1150 °C, no anneal

# Oxidized at 1150 °C, no anneal
SAMPLE ox1150C {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 72000 TEMP 1150->1150 AMBIENT dry_O2 TIMING stated
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

1150 °C, argon 1300 °C, rapid cooling

# 1150 °C, argon 1300 °C, rapid cooling
SAMPLE ox1150C_Ar1300C_cool600 {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 72000 TEMP 1150->1150 AMBIENT dry_O2 TIMING stated
    EVENT Transfer KIND transfer TIMING assumed                  # Moved to a different furnace
    RUN RampUp DURATION 300 TEMP 25->1300 AMBIENT inert_Ar TIMING assumed VARY duration 60..1800
    RUN ArgonAnneal0 DURATION 300 TEMP 1300->1300 AMBIENT inert_Ar TIMING range VARY duration 60..600
    RUN CoolRapid DURATION 50 TEMP 1300->800 AMBIENT inert_Ar TIMING bound VARY duration 12.5..50
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

1150 °C, argon 1500 °C, rapid cooling

# 1150 °C, argon 1500 °C, rapid cooling
SAMPLE ox1150C_Ar1500C_cool600 {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 72000 TEMP 1150->1150 AMBIENT dry_O2 TIMING stated
    EVENT Transfer KIND transfer TIMING assumed                  # Moved to a different furnace
    RUN RampUp DURATION 300 TEMP 25->1500 AMBIENT inert_Ar TIMING assumed VARY duration 60..1800
    RUN ArgonAnneal0 DURATION 300 TEMP 1500->1500 AMBIENT inert_Ar TIMING range VARY duration 60..600
    RUN CoolRapid DURATION 70 TEMP 1500->800 AMBIENT inert_Ar TIMING bound VARY duration 17.5..70
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

1150 °C, argon 1500 °C, standard cooling

# 1150 °C, argon 1500 °C, standard cooling
SAMPLE ox1150C_Ar1500C_cool70 {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 72000 TEMP 1150->1150 AMBIENT dry_O2 TIMING stated
    EVENT Transfer KIND transfer TIMING assumed                  # Moved to a different furnace
    RUN RampUp DURATION 300 TEMP 25->1500 AMBIENT inert_Ar TIMING assumed VARY duration 60..1800
    RUN ArgonAnneal0 DURATION 300 TEMP 1500->1500 AMBIENT inert_Ar TIMING range VARY duration 60..600
    RUN CoolStandard DURATION 600 TEMP 1500->800 AMBIENT inert_Ar TIMING bound VARY duration 600..2400
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

Oxidized at 1300 °C, no anneal

# Oxidized at 1300 °C, no anneal
SAMPLE ox1300C {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 2400 TEMP 1300->1300 AMBIENT dry_O2 TIMING range VARY duration 2400..4800
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

1300 °C, argon 1500 °C, rapid cooling

# 1300 °C, argon 1500 °C, rapid cooling
SAMPLE ox1300C_Ar1500C_cool600 {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 2400 TEMP 1300->1300 AMBIENT dry_O2 TIMING range VARY duration 2400..4800
    EVENT Transfer KIND transfer TIMING assumed                  # Moved to a different furnace
    RUN RampUp DURATION 300 TEMP 25->1500 AMBIENT inert_Ar TIMING assumed VARY duration 60..1800
    RUN ArgonAnneal0 DURATION 300 TEMP 1500->1500 AMBIENT inert_Ar TIMING range VARY duration 60..600
    RUN CoolRapid DURATION 70 TEMP 1500->800 AMBIENT inert_Ar TIMING bound VARY duration 17.5..70
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

1300 °C, argon 1500 °C, then argon 1300 °C

# 1300 °C, argon 1500 °C, then argon 1300 °C
SAMPLE ox1300C_Ar1500C_Ar1300C_cool600 {
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
    DOPING: n_type_1e16
    OFFCUT: 4_degrees
}
CONTROL_TRAJECTORY History {
    EVENT Clean KIND clean TIMING assumed                        # Clean and load
    RUN Oxidation DURATION 2400 TEMP 1300->1300 AMBIENT dry_O2 TIMING range VARY duration 2400..4800
    EVENT Transfer KIND transfer TIMING assumed                  # Moved to a different furnace
    RUN RampUp DURATION 300 TEMP 25->1500 AMBIENT inert_Ar TIMING assumed VARY duration 60..1800
    RUN ArgonAnneal0 DURATION 300 TEMP 1500->1500 AMBIENT inert_Ar TIMING range VARY duration 60..600
    RUN ArgonAnneal1 DURATION 300 TEMP 1300->1300 AMBIENT inert_Ar TIMING range
    RUN CoolRapid DURATION 50 TEMP 1300->800 AMBIENT inert_Ar TIMING bound VARY duration 12.5..50
    EVENT Metal KIND metallization TIMING assumed                # Aluminium electrodes
    EVENT CV KIND measurement TIMING assumed                     # C-V, high (1 MHz)-low method
}

The programs the predictor ran on these histories: experiments/sic_interface_states/one_way_anneal.isl.template, experiments/song_sic/oxide_growth_sic.isl.template.

The model itself, experiments/sic_interface_states/one_way_anneal.isl.template, as it is written. Every equation the simulation uses is in this text; the runtime holds no physics.

# ==============================================================================
# ISL MODEL: INTERFACE STATES OF SiC/SiO2 THROUGH ARGON ANNEALS
#
# Two parts. One is stable. The other is removable: a hold in argon takes it down to a level
# set by the anneal temperature, and it does not come back at a lower temperature. The density
# at each energy below the conduction band is the stable part plus the removable part, each
# with its own weight at that energy.
#
# The form was chosen among three (decay to nothing, a reversible equilibrium, a one-way level)
# by scripts/run_sic_interface_states.py, on the samples oxidized at 1150 C. How fast the level
# is reached is not resolved by any source we hold (the anneals are "1 to 10 min"), so the rate
# is a hypothesis: fast enough to finish within a minute.
# Fields in {{braces}} are filled by the runner.
# ==============================================================================
SAMPLE {sample_id} {{
    FAMILY: SiC_SiO2
    ORIENTATION: 4H_SiC_Si_face
    CLEAN: not_stated
}}
STATE removable {{ UNIT: 1; INITIAL: 1; MIN: 0; MAX: 1 }}
CONTROL T {{ UNIT: K }}
CONTROL p_Ar {{ UNIT: atm }}
PARAMETER log_level {{ VALUE: -0.3; UNIT: 1; STATUS: fitted }}
PARAMETER E_level {{ VALUE: 1.0; UNIT: eV; STATUS: fitted }}
PARAMETER log_S_0p3 {{ VALUE: 11.8; UNIT: 1; STATUS: fitted }}
PARAMETER log_S_0p4 {{ VALUE: 11.5; UNIT: 1; STATUS: fitted }}
PARAMETER log_S_0p5 {{ VALUE: 11.2; UNIT: 1; STATUS: fitted }}
PARAMETER log_R_0p3 {{ VALUE: 12.4; UNIT: 1; STATUS: fitted }}
PARAMETER log_R_0p4 {{ VALUE: 12.3; UNIT: 1; STATUS: fitted }}
PARAMETER log_R_0p5 {{ VALUE: 12.1; UNIT: 1; STATUS: fitted }}
PARAMETER settle_rate {{ VALUE: 0.1; UNIT: s^-1; STATUS: hypothesis }}
PARAMETER way {{ VALUE: {way}; UNIT: 1; STATUS: constant }}
PARAMETER soft {{ VALUE: 1.0e-3; UNIT: 1; STATUS: constant }}
PARAMETER T_ref {{ VALUE: 1573.15; UNIT: K; STATUS: constant }}
PARAMETER D_unit {{ VALUE: 1.0; UNIT: cm^-2 eV^-1; STATUS: constant }}
PARAMETER p_ref {{ VALUE: 1.0; UNIT: atm; STATUS: constant }}
PARAMETER ln10 {{ VALUE: 2.302585092994046; UNIT: 1; STATUS: constant }}
PARAMETER k_B {{ VALUE: 8.617333262e-5; UNIT: eV K^-1; STATUS: constant }}
# The level the anneal temperature allows, as a fraction of the as-oxidized removable part.
QUANTITY raw_level {{ UNIT: 1; EQUATION: raw_level = exp(ln10 * log_level) * exp((E_level / k_B) * (1 / T - 1 / T_ref)) }}
# Never more than the as-oxidized amount.
QUANTITY level {{ UNIT: 1; EQUATION: level = (raw_level + 1 - sqrt((raw_level - 1) * (raw_level - 1) + soft * soft)) / 2 }}
# way = 1: one-way (acts only while above the level). way = 0: reversible (moves to the level from either side).
QUANTITY above {{ UNIT: 1; EQUATION: above = (1 - way) + way * (1 + (removable - level) / sqrt((removable - level) * (removable - level) + soft * soft)) / 2 }}
CONTROL_TRAJECTORY {trajectory_id} {{
{runs}
}}
OPERATOR F_ArgonAnneal : Relaxation {{
    EQUATION: d(removable)/dt = -settle_rate * (p_Ar / p_ref) * above * (removable - {target})
}}
OBSERVATION_MODEL Obs_0p3 {{ TARGET: Dit_at_0p3eV_below_Ec; UNIT: cm^-2 eV^-1; EQUATION: Dit_at_0p3eV_below_Ec = D_unit * (exp(ln10 * log_S_0p3) + removable * exp(ln10 * log_R_0p3)) }}
OBSERVATION_MODEL Obs_0p4 {{ TARGET: Dit_at_0p4eV_below_Ec; UNIT: cm^-2 eV^-1; EQUATION: Dit_at_0p4eV_below_Ec = D_unit * (exp(ln10 * log_S_0p4) + removable * exp(ln10 * log_R_0p4)) }}
OBSERVATION_MODEL Obs_0p5 {{ TARGET: Dit_at_0p5eV_below_Ec; UNIT: cm^-2 eV^-1; EQUATION: Dit_at_0p5eV_below_Ec = D_unit * (exp(ln10 * log_S_0p5) + removable * exp(ln10 * log_R_0p5)) }}
EVALUATION_CONTRACT {contract_id} {{
    FROZEN: true
    DATASET: data/derived/kobayashi_2017_sic_dit.csv
    DATASET_DIGEST: sha256:{dataset_sha256}
    SPLIT: {split}
{targets}
}}

Our simulation against the paper

21 within of 21 numbers. Tolerance is stated per row; nothing was fitted for this page.

Aligning the simulation: what the unstated things would have to be

Every step whose timing or temperature the paper leaves open carries a range in the history above (VARY). scripts/align_study.py searched those ranges against the paper's numbers through the same predictor, first one unknown at a time, then all together. No parameter a program declares fitted was touched. The unknowns were chosen on ox1150C, ox1150C_Ar1300C_cool600, ox1150C_Ar1500C_cool600, ox1150C_Ar1500C_cool70 only; every other number is a test they played no part in.

Together: RampUp duration 300 s (was 300); ArgonAnneal0 duration 60 s (was 300); CoolRapid duration 1× (was 1); CoolStandard duration 2100 s (was 600).

As reproduced: 21 of 21 within tolerance, median error 4.7%. Aligned: 21 of 21 within tolerance, median error 4.6%. On the held-out samples only: 9 of 9 within tolerance, median error 4.7% as reproduced, 9 of 9 within tolerance, median error 4.6% aligned.

Choices of program or material property, each scored on its own and on top of the aligned history:

What was searched. Five things the paper gives only as ranges or not at all: the ramp to the anneal temperature in the anneal furnace (not described; 1 to 30 min, written as a new step, RampUp), the argon anneal time (“1 - 10 min”), the 1300 °C oxidation time (“40 min or 1h 20 min”), and the two cooling rates (“< 70” and “> 600 °C/min”, searched from 70 down to 17.5 °C/min and from 600 up to 2400 °C/min). The time of the second anneal of the twice-annealed sample is also “1 - 10 min”, but that sample is a test sample, so it was left at 5 min. Exponential cooling at the bound rate and a furnace already hot at loading are variants.

Read this as an upper bound. The unknowns were chosen on the four samples oxidized at 1150 °C (12 values), the same samples the program's eight fitted numbers were fitted on (evaluations/sic_interface_states_report.json). The three samples oxidized at 1300 °C (9 values) were not used in that fit or here, and are the test; they carry the same argon anneals at 1500 °C, so they test the oxidation temperature, which the program does not read.

Nothing in the history moves the numbers. On the chosen samples every unknown is flat: 12 of 12 within tolerance at every value searched, the median error between 4.2% and 4.5%. The anneal time does not matter between 1 and 10 min because the program's settling rate (0.1 s^-1, a hypothesis, not a fit) finishes the reduction in under a minute. The oxidation time and the cooling rates are not read: the as-oxidized level is a constant, and the reduction is one-way, so a slower cooling only lets it go on a little longer (median 4.5% at 600 s, 4.2% at 2400 s for the standard cooling). The combined search, which breaks ties on median error, settles on a 1 min anneal and a standard cooling of 35 min (20 °C/min) with nothing else changed. Aligned: 21 of 21, median error 4.6% (4.7% as reproduced); on the 1300 °C samples 9 of 9 both ways, median 4.7% to 4.6%. The two variants give 21 of 21 as well (median 4.7% alone, 4.6% and 4.8% on the aligned history). None of these differences is evidence.

The search found a fault in the program instead. With a ramp of 9.7 min or longer from room temperature in argon, the predictor refuses every annealed sample: the state removable reaches 1.00 and is reported as outside its bounds [0, 1]. At low temperature the program's level is softly capped just below 1, and a long time there lets the solver step over it. At 9.7 min 9 of the 12 chosen values are still simulated, at 17 and 30 min only the 3 as-oxidized ones. The search did not choose those values only because it counts values within tolerance, not values refused; it would have been wrong for it to choose them on median error.

9 within of 9 numbers with the aligned settings (the samples used to choose the unknowns are left out).

This paper cannot discriminate between histories, so alignment cannot test the program here. What the data do show and the program does not have is the cooling rate: at 0.4 and 0.5 eV the 1500 °C samples differ by 18 and 32% between standard and rapid cooling (4.25 against 3.60, 2.19 against 1.66 × 10^11 cm^-2 eV^-1), more states after the slow cooling; after argon at 1300 °C the slow-cooled sample (not among the cases here) has fewer (1.18 against 1.65 × 10^12 at 0.4 eV). The program gives each pair the same value. Because each cooling rate is also a different furnace, that difference cannot be assigned to the cooling alone.

The program's anneal clock is a hypothesis (0.1 s^-1). A sample annealed for less than a minute, or an isothermal series in time, would be needed to test it; this paper has neither.

What our language and ontology cannot yet say

• A cooling rate as a statement (COOL RATE <K/min> TO <T> [BOUND upper|lower]), so “< 70 °C/min” is written as the paper gives it rather than as an invented end temperature and duration. The rate also needs its own furnace, so a cooling difference is not silently also a furnace difference.
• A furnace or equipment field on a RUN (FURNACE resistive | rf_induction), and the residual oxygen it is known for (under 1 ppm for one, not reported for the other). Here the two cooling rates are two furnaces.
• A transfer through air as a RUN-like step with its own gas and time (TRANSFER AMBIENT air DURATION …): the Transfer here is an EVENT, so any exposure of the interface to room air between furnaces cannot reach a program.
• OFFCUT read by programs (4° off-axis here, on-axis in Song 2004), and the measurement method on an observable (1 MHz high-low counts fewer states than a 100 MHz reference, the paper's Figure 4).
• For the predictor: a small tolerance on a state's declared bounds, or a hard clamp, so that a long low-temperature hold does not refuse a history at exactly 1.00.

With heating and cooling transitions

No paper we hold states how its wafers heated up or cooled down. The project owner specified scenarios for one Vitkavage wafer (isl_harness/vitkavage_transitions.py); isl_harness/thermal_transitions.py applies the same numbers to every sample here: a heat-up in nitrogen before the first furnace step (a hot push of 1 to 5 min, or a cold-furnace ramp of 60 to 120 min), the owner's radiative cooling knots after the last furnace step or after the paper's own pull (fast, or every time multiplied by 5), and ramps around an anneal at 300 to 500 °C. The paper's own steps are kept as written. All transitions are assumptions, written in the history with TIMING assumed.

The paper's own conclusions

What we would like Emily to check

1. Is five minutes a fair single value for the argon anneal, or does the reduction depend on time within 1 to 10 minutes?
2. Between oxidation and the argon anneal the sample saw air at room temperature. Does that matter for this interface?
3. With the 1 MHz high-low method on SiC, how much of the state density is missed, and should we compare against the 100 MHz curves instead?