Communication-Efficient Pilot Estimation for Non-Randomly Distributed Data in Diverging Dimensions
针对分布式数据非随机分布和高维问题，提出通信高效的CEP估计方法，理论证明其收敛速度和渐近正态性，并通过模拟和真实数据验证有效性。
The communication-efficient surrogate likelihood (CSL) framework (Jordan, Lee, and Yang) is notable for handling massive or distributed datasets. The CSL methods use the first machine as the central one for optimization with its data and assume a fixed dimension for statistical properties. However, CSL may not suit non-randomly or heterogeneously distributed data and limit its applicability to diverging- or high-dimensional datasets. To address these issues, we propose a communication-efficient pilot (CEP) estimation strategy. This involves pilot sampling on each machine to create a pilot sample dataset and using a new pilot sample-based surrogate loss to approximate the global one, with the minimizer termed the CEP estimator. We rigorously investigate theoretical properties of the CEP estimator including its convergence rate, reaching the global rate pnN, and its asymptotic normality when the dimension pn diverges with the pilot sample size r and pn<n. Additionally, we extend CEP to high-dimensional cases (pn>n) and propose a regularized version of CEP (CERP). We establish non-asymptotic error bounds for the CERP estimator with Lasso penalty (CERP-Lasso) and provide convergence rates and asymptotic normality for the CERP estimator with adaptive Lasso penalty (CERP-aLasso) under generalized linear models. Extensive synthetic and real datasets demonstrate the effectiveness of our approaches. Supplementary materials for this article are available online.