When you build a predictive model in informetrics or scientometrics, you rarely start with a clean set of variables. Suppose you want to predict how many citations a paper will receive. You might have a dozen candidate predictors on hand: number of authors, number of references, page count, journal impact factor, the first author’s h-index, open access status, and more. Some of these genuinely drive citations. Others add noise. The challenge is deciding which variables belong in your model and which should be left out. Stepwise regression is one of the oldest and most widely used answers to that question, and understanding how it works, and where it goes wrong, is essential for anyone analysing bibliometric data.
Table of Contents
What is stepwise regression?
Stepwise regression is a method for fitting a regression model by automatically selecting a subset of independent variables from a larger pool of candidates. Instead of you deciding upfront exactly which predictors to keep, the algorithm builds the model one variable at a time. At each step, it adds or removes a single variable based on a statistical rule, then refits the model and checks again. This continues until no further change improves the model according to the chosen criterion.
The core idea is to find a parsimonious model, that is, a model that explains the outcome well while using as few predictors as possible. Running a regression with many variables, including irrelevant ones, produces a needlessly complex model that is hard to interpret and prone to error. Stepwise selection tries to trim that list down to a simple, interpretable set of variables that actually carry signal.
A useful way to picture the process: imagine your full set of candidate predictors as a list. The algorithm does not test every possible combination of variables (that would be computationally expensive with many predictors). Instead, it “steps” through the list, evaluating one variable at a time and making a local decision at each step. This greedy, step-by-step nature is exactly why it is called stepwise regression. The widely used algorithm behind it was first proposed by Efroymson in 1960.
The statistical criteria that drive the decisions
At every step, the algorithm needs a rule to decide whether a variable should enter or leave the model. Three criteria dominate practice.
The first is the p-value. A variable is added if its coefficient is statistically significant below a chosen entry threshold, and removed if its p-value rises above an exit threshold. This is intuitive but has a hidden cost we will return to later.
The second is the Akaike Information Criterion (AIC). AIC penalises the model’s likelihood by the number of parameters it includes, so models that fit well using fewer variables are preferred. After fitting an initial model, the AIC is computed for each submodel that would result from adding or removing a feature, and the algorithm moves to whichever option gives the best AIC. The process repeats until no change improves it.
The third is the Bayesian Information Criterion (BIC). BIC is similar to AIC but applies a heavier penalty for the number of parameters, especially as the sample size grows. Both AIC and BIC are penalised-likelihood criteria that try to balance goodness of fit against the risk of overfitting. In practice, BIC tends to select smaller models than AIC.
Types of stepwise regression
Stepwise regression comes in three main variants. They differ in where they start and in the direction they move.
Forward selection
Forward selection begins with an empty model containing no predictor variables. Each candidate variable is tested one at a time, and the one that gives the greatest improvement to the model (for example, the lowest p-value or the best AIC) is added first. The model is refitted, and the process repeats: at each stage, the most significant remaining variable is brought in. This continues until no remaining variable can add anything statistically significant to the model.
The appeal of forward selection is that it starts small and grows. This makes it well suited to situations where you have many candidate variables but suspect that only a handful matter. A key drawback, though, is that a variable added early stays in the model permanently, even if later additions make it redundant.
Backward elimination
Backward elimination is the mirror image. It starts with the full model that includes every candidate predictor and removes variables one at a time. At each step, the least useful variable, often the one with the highest p-value or the one whose removal most improves the adjusted R-squared, is deleted. The model is refitted and the process repeats until no variable can be removed without a statistically significant loss of fit.
Backward elimination has a practical limitation worth noting. Because it begins with the full model, it requires the dataset to support estimating every variable at once. It is generally recommended only with large sample sizes, where the number of observations comfortably exceeds the number of predictors. With many candidate variables and few observations, the starting full model may be unstable or impossible to fit.
Bidirectional (combined) stepwise
The third variant combines both directions and is what people most often mean by “stepwise regression.” It adds the most significant variable as in forward selection, but at each step it also checks whether any variable already in the model has become insignificant because of the new addition. If so, that variable is removed. The process of adding and removing continues until no further addition or deletion improves the model.
This is a meaningful improvement over plain forward selection. In a combined procedure, a variable might be added at step 2, dropped at step 5, and added again at step 9. Because variables that lose their usefulness can be discarded, the final model is less likely to retain predictors that only mattered early on. The trade-off is that the procedure runs more comparisons at each step.
Application in informetrics and scientometrics
Stepwise regression is a natural fit for bibliometric prediction problems, because these datasets often contain a large array of explanatory variables and no clear theory about which ones matter most. Citation prediction is the classic example.
Researchers have used multiple regression to predict citation counts from features such as the number of authors, number of pages, and number of references. In one analysis of big data research, the number of citations was modelled using author count, page count, and reference count, with the significance of each predictor tested individually. When a study like this starts with many candidate indicators, a stepwise procedure can help identify which subset actually contributes to the prediction.
The scope of candidate variables in scientometrics can be wide. Studies predicting the time to a paper’s first citation have drawn on indicators grouped into categories such as article characteristics, first-author impact, the highest h-index author’s citations, self-citations, reference list factors, and journal quality indicators. With so many possible predictors, manually deciding which to keep is difficult, and an automated selection method offers a structured starting point.
Patent analysis follows the same pattern. Researchers have explained patent citation frequency using feature values such as the numbers of inventors, classifications, pages, figures, tables, and claims drawn from patent records. Across these applications, the common thread is that the analyst faces more candidate variables than the model should sensibly hold, which is precisely the problem stepwise regression was designed to address.
Limitations and the risk of overfitting
Stepwise regression is convenient, but it carries serious statistical risks that every analyst should understand before relying on it.
The most important is overfitting. Because the algorithm searches through many models to find the combination that fits the data best, it tends to capture quirks specific to the sample rather than the true underlying relationship. As a result, a stepwise model often fits much better on the data used to build it than on new, out-of-sample data. In extreme cases, stepwise procedures have produced statistically significant models even when run on pure random numbers. This is a sobering reminder that significance, on its own, does not guarantee a meaningful model.
A closely related problem is the inflation of false positives. Stepwise selection makes its decisions using many repeated hypothesis tests, but the p-values it reports are not adjusted for all that testing. This leads to over-selection of features and false positive findings, a problem made worse when predictors are highly correlated. The reported coefficients and their uncertainty are therefore overly optimistic, because they ignore the selection process that produced them.
Multicollinearity compounds these issues. When candidate predictors are strongly correlated with each other, which is common in bibliometric data where many indicators measure related aspects of impact, stepwise methods can behave erratically. The procedure may pick one variable over an equally valid alternative for reasons that have little to do with substance. Model instability is a visible symptom: small changes in the data can sometimes lead to a completely different set of selected variables.
There is also a conceptual limitation. Stepwise regression hands the choice of variables to a computer program, without any consideration for what the variables actually measure or how they fit into the theoretical framework of the study. A variable might be dropped because it narrowly missed a significance threshold, even though theory and prior research strongly suggest it belongs. For this reason, it is usually more appropriate to let theory and previous findings guide which variables to include, using stepwise output as a supporting tool rather than the final word.
How to use it more safely
None of this means stepwise regression should never be used. It remains genuinely useful when the number of candidate variables is large and exhaustive search of all possible models becomes computationally too expensive. The key is to use it carefully.
First, prefer information criteria such as AIC or BIC over raw p-value thresholds, since they explicitly penalise complexity. Second, always validate the chosen model on a separate test set; splitting your data into training and test portions is essential when the goal is prediction. A large gap between training and test performance is a clear warning sign of overfitting. Third, treat the selected variables as a hypothesis to be examined with domain knowledge, not as a definitive answer. In scientometrics in particular, the most reliable models combine the analyst’s expertise with careful attention to the structure of the bibliometric data.
What do you think? If you were building a model to predict the citation impact of papers in your own field, which variables would you trust your theoretical knowledge to choose, and which would you be willing to let an automated stepwise procedure decide? And given the overfitting risks, how much weight should bibliometric studies place on models whose variables were selected by an algorithm rather than by theory?
References
- https://en.wikipedia.org/wiki/Stepwise_regression
- https://quantifyinghealth.com/stepwise-selection/
- https://link.springer.com/article/10.1186/s40537-018-0143-6
- https://bookdown.org/max/FES/greedy-stepwise-selection.html
- https://arxiv.org/pdf/1910.12389
- https://www.spcforexcel.com/knowledge/root-cause-analysis/forward-selection-regression-and-backward-elimination-regression/
- https://courses.lumenlearning.com/introstats1/chapter/model-selection/
- https://www.statisticalaid.com/stepwise-regression/
- https://link.springer.com/article/10.1186/s40537-017-0088-1
- https://link.springer.com/article/10.1007/s11192-025-05237-x
- https://www.numberanalytics.com/blog/stepwise-regression-techniques-unveiled
- https://www.statisticssolutions.com/stepwise-regression-what-is-it-and-should-you-use-it/
- https://towardsdatascience.com/model-selection-in-linear-regression/
Leave a Reply