Privacy Protection and Statistical Efficiency Trade-Off for Federated Learning

References

• Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L (2016) Deep learning with differential privacy. Proc. 2016 ACM SIGSAC Conf. Comput. Commun. Security CCS ‘16 (Association for Computing Machinery, New York), 308–318.Google Scholar
• Arjevani Y, Shalev-Shwartz S, Shamir O (2016) On lower and upper bounds in smooth and strongly convex optimization. J. Machine Learn. Res. 17(126):1–51.Google Scholar
• Bach F, Moulines E (2013) Non-strongly-convex smooth stochastic approximation with convergence rate o(1/n). Proc. 26th Internat. Conf. Neural Information Processing Systems NIPS’13, vol. 1 (Curran Associates Inc., Red Hook, NY), 773–781.Google Scholar
• Bi X, Shen X (2023) Distribution-invariant differential privacy. J. Econom. 235(2):444–453.Crossref, Google Scholar
• Burden R, Faires JD (2010) Numerical Analysis, 9th ed. (Cengage Learning, Boston).Google Scholar
• Cai J, Liu Y, Liu X, Li J, Zhuang H (2023) Privacy-preserving federated cross-domain social recommendation. Trustworthy Federated Learn. First Internat. Workshop FL 2022 Held Conjunction IJCAI 2022 Revised Selected Papers (Springer International Publishing, New York), 144–158.Google Scholar
• Chen X, Liu W, Zhang Y (2022) First-order Newton-type estimator for distributed estimation and inference. J. Amer. Statist. Assoc. 117(540):1858–1874.Crossref, Google Scholar
• Chen X, Lee JD, Tong XT, Zhang Y (2020) Statistical inference for model parameters in stochastic gradient descent. Ann. Statist. 48(1):251–273.Crossref, Google Scholar
• Chen H, Zhu T, Zhang T, Zhou W, Yu PS (2023) Privacy and fairness in federated learning: On the perspective of tradeoff. ACM Comput. Survey 56(2):1–37.Crossref, Google Scholar
• Collins L, Hassani H, Mokhtari A, Shakkottai S (2022) FedAvg with fine tuning: Local updates lead to representation learning. Oh AH, Agarwal A, Belgrave D, Cho K, eds. Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY), 10572–10586.Crossref, Google Scholar
• Devlin J, Chang M-W, Lee K, Bert KT (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. Preprint, submitted October 11, https://arxiv.org/abs/1810.04805v1.Google Scholar
• Dong J, Roth A, Su WJ (2022) Gaussian differential privacy. J. Roy. Statist. Soc. Ser. B Statist. Methodology 84(1):3–37.Crossref, Google Scholar
• Dwork C, Lei J (2009) Differential privacy and robust statistics. Proc. Forty-First Annu. ACM Sympos. Theory Comput. (Association for Computing Machinery, New York), 371–380,Google Scholar
• Firdaus M, Larasati HT, Rhee K-H (2022) A secure federated learning framework using blockchain and differential privacy. 2022 IEEE 9th Internat. Conf. Cyber Security Cloud Comput. CSCloud (IEEE, Piscataway, NJ), 18–23.Google Scholar
• Fu J, Hong Y, Ling X, Wang L, Ran X, Sun Z, Wang WH, Chen Z, Cao Y (2024) Differentially private federated learning: A systematic review. Preprint, submitted May 14, https://arxiv.org/abs/2405.08299.Google Scholar
• Geiping J, Bauermeister H, Dröge H, Moeller M (2020) Inverting gradients—How easy is it to break privacy in federated learning? Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds. Adv. Neural Inform. Processing Systems, vol. 33 (Curran Associates, Inc., Red Hook, NY), 16937–16947.Google Scholar
• Geyer RC, Klein T, Nabi M (2017) Differentially private federated learning: A client level perspective. Preprint, submitted December 20, https://arxiv.org/abs/1712.07557v1.Google Scholar
• Gitman I, Lang H, Zhang P, Xiao L (2019) Understanding the role of momentum in stochastic gradient methods. Proc. 33rd Conf. Neural Inform. Processing Systems (Curran Associates, Inc., Red Hook, NY).Google Scholar
• Guo Y, Liu F, Zhou T, Cai Z, Xiao N (2023) Privacy vs. efficiency: Achieving both through adaptive hierarchical federated learning. IEEE Trans. Parallel Distributed Systems 34:1331–1342.Crossref, Google Scholar
• Kim M, Günlü O, Schaefer RF (2021) Federated learning with local differential privacy: Trade-offs between privacy, utility, and communication. ICASSP 2021–2021 IEEE Internat. Conf. Acoustics Speech Signal Processing ICASSP (IEEE, Piscataway, NJ), 2650–2654.Google Scholar
• Li J, Shao Y, Wei K, Ding M, Ma C, Shi L, Han Z, Vincent Poor H (2022) Blockchain assisted decentralized federated learning (BLADE-FL): Performance analysis and resource allocation. IEEE Trans. Parallel Distributed Systems 33(10):2401–2415.Crossref, Google Scholar
• Liu X, Kong W, Oh S (2022) Differential privacy and robust statistics in high dimensions. Proc. Thirty Fifth Conf. Learn. Theory, Proceedings of the Machine Learning Research, vol. 178 (PMLR, New York), 1167–1246.Google Scholar
• Luo J, Xu Z (2023) Differential privacy statistical inference for a directed graph network model with covariates. Preprint, submitted December 8, https://arxiv.org/abs/2312.04903.Google Scholar
• McMahan B, Moore E, Ramage D, Hampson S, Agüera y Arcas B (2017) Communication-efficient learning of deep networks from decentralized data. Artificial Intelligence and Statistics, vol. 54 (PMLR, New York), 1273–1282.Google Scholar
• Mironov I (2017) Rényi differential privacy. 2017 IEEE 30th Comput. Security Foundations Sympos. CSF (IEEE, Piscataway, NJ).Google Scholar
• Noble M, Bellet A, Dieuleveut A (2022) Differentially private federated learning on heterogeneous data. Internat. Conf. Artificial Intelligence Statist., vol. 151 (PMLR, New York), 10110–10145.Google Scholar
• Nocedal J, Wright SJ (2006) Numerical Optimization, 2nd ed. (Springer, New York).Google Scholar
• Pichler G, Romanelli M, Vega LR, Piantanida P (2023) Perfectly accurate membership inference by a dishonest central server in federated learning. IEEE Trans. Dependable Secure Comput. 21(4):4290–4296.Google Scholar
• Polyak BT, Juditsky AB (1992) Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30(4):838–855.Crossref, Google Scholar
• Qi H, Wang F, Wang H (2025) Privacy protection and statistical efficiency trade-off for federated learning. https://doi.org/10.1287/ijoc.2024.0554.cd, https://github.com/INFORMSJoC/2024.0554.Google Scholar
• Song L, Wang H, Zhang G, Yu S (2023) FedInf: Social influence prediction with federated learning. Neurocomputing 548:126407.Crossref, Google Scholar
• Wang Q, Yan T, Jiang B, Leng C (2022) Two-mode networks: Inference with as many parameters as actors and differential privacy. J. Machine Learn. Res. 23(292):1–38.Google Scholar
• Wang Y, Su Z, Pan Y, Luan TH, Li R, Yu S (2024) Social-aware clustered federated learning with customized privacy preservation. IEEE/ACM Trans. Networking 32(5):3654–3668.Crossref, Google Scholar
• Wasserman L, Zhou S (2010) A statistical framework for differential privacy. J. Amer. Statist. Assoc. 105(489):375–389.Crossref, Google Scholar
• Watkins W, Wang H, Bae S, Tseng H-H, Cha J, Chen SY-C, Yoo S (2024) Quantum privacy aggregation of teacher ensembles (QPATE) for privacy-preserving quantum machine learning. ICASSP 2024–2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, Piscataway, NJ), 6875–6879.Google Scholar
• Wei K, Li J, Ma C, Ding M, Chen W, Wu J, Tao M, Poor HV (2023) Personalized federated learning with differential privacy and convergence guarantee. IEEE Trans. Inform. Forensics Security 18:4488–4503.Crossref, Google Scholar
• Wei K, Li J, Ding M, Ma C, Yang HH, Farokhi F, Jin S, Quek TQS, Poor HV (2020) Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans. Inform. Forensics Security 15:3454–3469.Crossref, Google Scholar
• Yousefpour A, Shilov I, Sablayrolles A, Testuggine D, Prasad K, Malek M, Nguyen J, et al. (2021) Opacus: User-friendly differential privacy library in PyTorch. Preprint, submitted September 25, https://arxiv.org/abs/2109.12298v1.Google Scholar
• Zhang Q, Ma J, Lou J, Xiong L, Jiang X (2020) Towards training robust private aggregation of teacher ensembles under noisy labels. 2020 IEEE Internat. Conf. Big Data Big Data (IEEE, Piscataway, NJ), 1103–1110,Google Scholar
• Zhang X, Kang Y, Chen K, Fan L, Yang Q (2023) Trading off privacy, utility, and efficiency in federated learning. ACM Trans. Intelligent Systems Tech. 14(6):98.Crossref, Google Scholar
• Zhao X, Cui Q, Li W, Ni W, Hossain E, Sheng QZ, Tao X, Zhang P (2025) Convergence-privacy-fairness trade-off in personalized federated learning. Trans. Machine Learn. Commun. Networking 3:246–262.Crossref, Google Scholar
• Zheng Q, Chen S, Long Q, Su W (2021) Federated f-differential privacy. Internat. Conf. Artificial Intelligence Statist. (PMLR, New York), 2251–2259.Google Scholar