记忆约束下的最优单遍非参数估计

Optimal One-Pass Nonparametric Estimation Under Memory Constraint

Journal of the American Statistical Association · 2022
被引 15
ABS 4

中文导读

针对流式数据中记忆有限的问题,提出基于惩罚正交基展开的单遍估计器,证明其在记忆约束下统计最优且记忆占用渐近最小,数值实验显示其效率接近可访问全部历史数据的非流式方法。

Abstract

For nonparametric regression in the streaming setting, where data constantly flow in and require real-time analysis, a main challenge is that data are cleared from the computer system once processed due to limited computer memory and storage. We tackle the challenge by proposing a novel one-pass estimator based on penalized orthogonal basis expansions and developing a general framework to study the interplay between statistical efficiency and memory consumption of estimators. We show that, the proposed estimator is statistically optimal under memory constraint, and has asymptotically minimal memory footprints among all one-pass estimators of the same estimation quality. Numerical studies demonstrate that the proposed one-pass estimator is nearly as efficient as its nonstreaming counterpart that has access to all historical data.

非参数回归流式数据统计效率记忆约束正交基展开