GLM-5.3-Flash FP8 Optimized Inference Support

[zzyu17]

As stated in docs/MODELS.md (line 46):

| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented

inference for GLM-5.3-Flash FP8 model is not yet supported in the current ds4 engine.

I tried simply patching ds4.c by routing GLM Q8_0 experts through the existing generic Metal ID kernels and adding Q8_0 coverage to the F32 reference path (see https://github.com/zzyu17/ds4/commit/61b0d30c30d05cd118335f60e044a8f772225c01 for details), which gives the below results for a GLM-5.3-Flash Q8_0 GGUF on M3 Ultra 512G.

The performance is quite good, with ~390 token/s prefill and ~21 token/s generation. However, an optimized (Metal) inference kernel for GLM-5.3-Flash FP8 model should perform even better, compared to the generic Metal kernel patch.

So, looking for the optimized (Metal) inference support for GLM-5.3-Flash FP8 model, eagerly!

[edwardtian]

@zzyu17 Are you using GLM-5.3-Flash-FP8.gguf ? I tried your patch, got:
ds4: tensor output_norm.weight has type bf16, expected f32

Thanks in advance.

[zzyu17]

No, I don't use the official GLM-5.3-Flash-FP8.gguf from antirez/glm-5.3-flash-gguf on Hugging Face. I convert from the official zai-org/GLM-5.3-Flash safetensors to get the Q8_0 GGUF.

From my experience I can tell it may be, most probably, related to my patch around line 15308 (i.e., the glm_routed_moe_dot_f32 function). Another small patch around there or an extra conversion of your GGUF to make the weights match the expected float type may be a work around, because it's just all about the engine refusing an unmatched float type of tensor weights.

[zzyu17]

@edwardtian I have added another commit for supporting Q8_0 GGUF conversion from safetensors, which may help you with the model. See zzyu17@27809d0.