In this study, we develop an intelligence device to apply speech processing function in an Internet of Vehicles (IoV). The voice-based interactions will improve drive safety and in-time awareness of the vehicle status. This interaction can be achieved through speech recognition and response generation between the driver and the smart vehicle. Thus, the driver can focus on the driving. The proposed speech processing can be divided into three portions: (1) voice signal preprocessing, (2) speech recognition, and (3) speaker recognition. Firstly, speech signal pre-processing consists of five steps: sampling, pre-emphasis, frame, window function, and mel-frequency cepstral coefficients (MFCC), so as to be able to extract the characteristic parameters in the speech signal. Secondly, the speech model is built via the hidden Markov model (HMM), and the Viterbi algorithm is used to search the best sequence in the model to achieve the function of speech recognition. Finally, we use the Linde–Buzo–Gray (LBG) algorithm in vector quantization (VQ) to train for the speaker model, and then use cosine similarity to achieve the function of speaker recognition. The proposed speech processing function has been validated experimentally, and the experimental results demonstrate its feasibility for drivers to easily control the IoV system via voice-based command. In addition, the system distinguishes different speakers and provides the corresponding usage privileges, which will improve drive safety and in-time awareness of the general vehicle status.