Stanford's LABOR-LLM Predicts Career Transitions Using AI Logic
By treating professional histories as sequences of text, a new model forecasts worker job transitions with higher accuracy than traditional economic models.
Researchers at the Stanford Graduate School of Business, led by Professor Susan Athey, have developed LABOR-LLM, a model designed to predict a worker's next occupation based on their professional history. The system represents a shift in how labor economists analyze career trajectories, moving from rigid numerical categories to the flexible semantic processing of generative AI.
To build the model, the team converted tabular career survey data into text files that resemble resumes. By applying the "next-token prediction" logic used by Large Language Models (LLMs), LABOR-LLM treats a career as a sequence of words where each "token" is a job. This approach allows the system to calculate transition probabilities with greater precision than previous discrete-choice models. According to the research, LABOR-LLM outperforms prior methods in predicting both granular next occupations and general labor force status, such as whether a worker will remain employed or leave the workforce.
The Shift from Codes to Semantics
For decades, traditional labor economics relied on categorical, discrete-choice prediction models. While effective for simple datasets, these models often struggled with the high-dimensional nature of occupational histories—the vast and complex variety of paths a professional might take.
LABOR-LLM solves this by leveraging the semantic meaning inherent in job titles. The researchers found that predictive performance declines significantly when English language occupational titles are replaced with unique numerical codes. This suggests that the model is not merely tracking patterns in data, but is utilizing the underlying linguistic relationships between different roles to understand how one job naturally leads to another.
Implications for Workforce Planning
This breakthrough demonstrates that LLMs can encode complex world knowledge regarding occupational relationships. For the industry, this provides a sophisticated tool for career counseling and workforce planning, allowing for more accurate forecasting of how talent moves across sectors. For policymakers, it offers a data-driven method to design employment policies that align with actual professional movement patterns.
Furthermore, the research indicates that model size is not the only factor in accuracy. The team discovered that fine-tuning smaller LLMs can actually surpass the performance of larger models when additional career data from different populations is integrated into the training set.
Future Outlook
As the model evolves, the focus will likely shift toward how these predictions can be integrated into real-time labor market tools. While the current results prove the efficacy of the "career-as-text" approach, the researchers have highlighted the critical importance of semantic data over numerical indexing. Future developments will likely explore how diverse global datasets can further refine the accuracy of these professional forecasts.