Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation
Source
Claudia Grosser, Maike Heuer, Denis Krompass, Thomas A. Runkler
Author
Claudia Grosser, Maike Heuer, Denis Krompass, Thomas A. Runkler
Date
Key takeaways · AI-distilled
Federated pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → trains one model across data that never leaves the machines holding it. The catch is comparison: clients participate unevenly and hold different amounts of data, so two runs are hard to rank at all.
The test design is clean: train a 16M-parameter transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → centrally and federated on identical client data, rank the models by pre-training perplexity, then ask which evaluation protocol reproduces that ranking.
fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → on GLUE broke the ranking in every variant tried, including head-only and reduced-data runs. Task-specific adaptation contributes enough of the score to mask differences in the underlying pre-trained model.
Just measuring next-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → prediction on GLUE text, with no fine-tuning at all, tracked pre-training perplexity closely. It is the cheaper protocol and the more faithful one.
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
If you're comparing federated pre-training runs, this shows that fine-tuning on downstream tasks like GLUE can give you a misleading ranking of model quality — checking intrinsic next-token perplexity is a more trustworthy signal.
Key quotes
“Our results show that downstream fine-tuning does not reliably preserve the pre-training ranking, whereas direct next-token prediction exhibits a strong correspondence with the pre-training test perplexity.”
“These findings suggest that downstream fine-tuning alone can be misleading when comparing federated pre-trained models, and that evaluation signals closer to the original pre-training objective deserve greater attention.”
