The rapid scaling of large language models (LLMs) has led to unprecedented computational costs and environmental impact. We address the problem of multi-objective optimization for LLM training, balancing model performance against resource usage (training time, energy, GPU-hours, carbon footprint). We survey and evaluate state-of-the-art techniques\u2014model pruning, quantization, knowledge distillation, neural architecture search (NAS), and hyperparameter tuning via evolutionary or reinforcement learning\u2014in terms of their trade-offs between accuracy (or loss) and efficiency. Using recent experimental data from public benchmarks (e.g. BERT fine-tuning on GLUE tasks, GPT-family training), we analyze how each method shapes the Pareto frontier of accuracy vs. cost. For example, static 8-bit quantization has been shown to cut energy use by ~29% with negligible accuracy loss, while structured pruning can speed up inference by ~63% for minor accuracy degradation. Advanced LLM compression methods achieve even larger gains: a multi-objective shift-add reparameterization method achieved over 80% reduction in memory and energy usage compared to full models. We include Pareto-plots (accuracy vs energy) to visualize these trade-offs (Figure 1\u20134). Overall, we find that multi-objective search (e.g. Bayesian optimization or genetic algorithms) can systematically identify configurations that lie on the Pareto-optimal front, enabling practitioners to choose models that best fit their constraints. Our paper highlights that, consistent with \u201cGreen AI\u201d principles, moderate sacrifices in accuracy can yield large efficiency gains, and we provide actionable recommendations for training sustainable LLMs.<\/p>\n
Article Link:<\/p>\n