Many regression models for categorical data have been introduced in various applied fields, motivated by different paradigms. But these models are difficult to compare because their specifications are not homogeneous. The first contribution of this paper is to unify the specification of regression models for categorical response variables, whether nominal or ordinal. This unification is based on a decomposition of the link function into an inverse continuous cdf and a ratio of probabilities. This allows us to define the new family of reference models for nominal data, comparable to the adjacent, cumulative and sequential families of models for ordinal data. We introduce the notion of reversible models for ordinal data that enables to distinguish adjacent and cumulative models from sequential ones. Invariances under permutations of categories are then studied for each family. The combination of the proposed specification with the definition of reference and reversible models and the various invariance properties leads to an in-depth renewal of our view of regression models for categorical data. Finally, a family of new supervised classifiers is tested on three benchmark datasets and a biological dataset is investigated with the objective of recovering the order among categories with only partial ordering information.