Question about EXACT MATCH ACC

[xyzCS]

Hello, your work has made amazing progress on the spider data set, LGESQL + ELECTRA achieves 75.1 on the dev set, my question is whether this result is beam acc or sql acc?

[rhythmcao]

It is sql acc (exact set match accuracy without values, but we use a small widely-used trick which returns the first prediction in the beam without execution error). Not Beam Acc (as long as one prediction in the beam is correct compared to the ground truth, we consider it correct). Acctually, the beam acc can be larger than 80% w.r.t exact set match without values.

[xyzCS]

Thank you very much😁