Transformer language models process text through contextual token representations coupled by learned attention. We formulate a transformer context as a finite interacting many-body representation system and define correlation functions, susceptibility-like quantities, response functions, and order parameters for its hidden states. Applying these observables first to GPT-2 benchmark prompt ensembles, we find that token-token correlations develop layer-dependent nonlocal structure and that compact correlation-derived observables distinguish different prompt ensembles substantially better than simple output-level statistics. Because the benchmark ensembles differ in content and construction, we complement this result with a matched within-task comparison of correct and incorrect mathematical reasoning trajectories generated for the same underlying problems. The correlation-derived observables retain above-chance predictive information in this more stringent setting and remain informative across GPT-2, Pythia-410M, and Qwen2.5 models, although their quantitative behavior is model dependent. Perturbation experiments further exhibit an approximately linear internal response and quadratic output Kullback–Leibler divergence in the verified small-perturbation regime. The results support the use of statistical-physics observables for representation-level analysis while showing that their quantitative behavior is model dependent.