Intel / article

Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation

Source: arxiv.org
Author: Claudia Grosser, Maike Heuer, Denis Krompass, Thomas A. Runkler
Date: Published Aug 3, 2026

Context

A benchmark taken after task-specific training may change the ranking researchers intended to measure. In this controlled study, researchers compared versions of a 16-million-parameter trained centrally or across distributed datasets, using identical client data. They established a reference ranking from prediction performance on the test set. Additional training on language-understanding tasks did not reliably preserve that ranking; directly testing next-token prediction on those tasks' text tracked it more closely. The result helps distinguish two evaluation questions: how well pretraining worked and how well a model performs after adaptation. The study specifically tests fidelity to the first ranking.

Key takeaways · AI-distilled

Federated pre-training trains one model across data that never leaves the machines holding it. The catch is comparison: clients participate unevenly and hold different amounts of data, so two runs are hard to rank at all.
The test design is clean: train a 16M-parameter centrally and federated on identical client data, rank the models by pre-training perplexity, then ask which evaluation protocol reproduces that ranking.
on GLUE broke the ranking in every variant tried, including head-only and reduced-data runs. Task-specific adaptation contributes enough of the score to mask differences in the underlying pre-trained model.
Just measuring next-token prediction on GLUE text, with no fine-tuning at all, tracked pre-training perplexity closely. It is the cheaper protocol and the more faithful one.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

Key quotes

“Our results show that downstream fine-tuning does not reliably preserve the pre-training ranking, whereas direct next-token prediction exhibits a strong correspondence with the pre-training test perplexity.”

“These findings suggest that downstream fine-tuning alone can be misleading when comparing federated pre-trained models, and that evaluation signals closer to the original pre-training objective deserve greater attention.”

Read the source arxiv.org