Driver fatigue is a major cause of traffic accidents. Automatic vision-based driver fatigue recognition is one of the most prospective commercial applications based on facial expression analysis technology. Generally, factors such as noise, illumination effects, image scaling, and redundant data affect the performance of facial expression recognition systems. In this paper, we have proposed an efficient algorithm, which is not only capable of working with multi-scale images but also able to overcome the mentioned obstacles. The proposed framework can be divided into three main phases. In the first step, the input image is converted into four sub-band images by applying a discrete wavelet transform, which preserves the important information of face image. Also, the original image is down-sampled to obtain the image of different sizes. Based on entropy analysis, each image is then further divided into a number of blocks classified as either informative or non-informative blocks. In the second step, the high variance features are selected in a zigzag manner using discrete cosine transform. In the final step, classifiers are trained and tested to accurately classify the expressions into seven generic expression classes. The empirical results suggest that the proposed framework not only effectively utilizes the multi-scale images but also outperforms other similar techniques in terms of classification accuracy rate.