Gradient-Based Optimization for Large Neural Network Models
Analysis
This paper addresses the computational challenges of optimizing nonlinear objectives using neural networks as surrogates, particularly for large models. It focuses on improving the efficiency of local search methods, which are crucial for finding good solutions within practical time limits. The core contribution lies in developing a gradient-based algorithm with reduced per-iteration cost and further optimizing it for ReLU networks. The paper's significance is highlighted by its competitive and eventually dominant performance compared to existing local search methods as model size increases.
Key Takeaways
- •Proposes a new gradient-based optimization algorithm for neural network surrogates.
- •The algorithm is designed to have a lower per-iteration cost.
- •The algorithm is specifically adapted to exploit the structure of ReLU networks.
- •The method's performance improves relative to other local search methods as model size increases.
“The paper proposes a gradient-based algorithm with lower per-iteration cost than existing methods and adapts it to exploit the piecewise-linear structure of ReLU networks.”