PROSODY MODELING FOR STORYTELLING STYLE SPEECH SYNTHESIS

PARAKRANT SARKAR
Masters (By Research) Student
Department of Computer Science and Engineering
Indian Institute Of Technology Kharagpur, West Bengal, INDIA
Email: parakrantsarkar@gmail.com

Objective:

The objective of this study is to developed story TTS systems in Hindi using syllables as basic unit.

We have developed two story TTS systems.
- The first system is developed using neutral TTS system and second is a unit selection based TTS system developed using story speech corpus. The first system uses the neutral TTS system to convert the neutral speech into story style speech. The conversion is carried out by our three proposed modules. These modules can be easily plugged into the neutral TTS system. The speech generated by the neutral TTS system is modified into story style using the three modules. The three modules designed are:
  1. Story-specific emotion detection (SSED) module
  2. Story-specific prosody generation (SSPG) module
  3. Story-specific prosody incorporation (SSPI) module.
- The Figure given displays synthesis framework of the first story TTS system. The Story-specific prosody generation (SSPG) module include prosody rules which are derived for five story-specific emotions (i.e., happy, fear, sad, anger, and neutral) by analyzing the difference in prosodic parameters of neutral and original storyteller speech utterances. The neutral TTS system with these designed modules can be able to generate story style speech.
- The second TTS system is a corpus based synthesis system. It uses unit section to appropriately select the units (i.e., syllable) and concatenate them (using signal processing techniques) to generate story style speech. The efficiency of the story TTS system depends on how good the linguistic, semantic and the prosodic information are extracted from the text, and how well these informations are incorporated into the synthesized speech. In corpus based synthesis, the story TTS system is build from the story corpus recorded by a professional human storyteller. Here, the objective is to enhance the synthesis quality of synthesized story style speech. To enhance the quality, we need to improve the prosody models of story TTS system. In view of this, prosody modeling frameworks are proposed for duration, intonation, intensity and pause pattern of story style speech.
The scope of the work is described are as follows:

Development of Hindi story TTS system using the neutral TTS system.
Development of Hindi story TTS system using a story corpus.
Modeling of story-specific pauses for story style speech synthesis
Prosody (i.e. duration, intonation, and intensity) modeling of the syllables based on story genre.

Story TTS systems

Wavefiles
neutral story speech story style speech synthesized from story TTS system developed using neutral TTS story style speech synthesized from story TTS system developed using story corpus

Story-specific pause prediction model

Wavefiles
no-pause-model default-pause-model
Proposed pause prediction models single-CART single-ELM single-FFNN single-SVM multi-model-CART multi-model-FFNN multi-model-ELM multi-model-SVM discourse-multi-model-CART discourse-multi-model-ELM discourse-multi-model-FFNN discourse-multi-model-SVM

Prosody Modeling for storytelling style speech synthesis using story genre

Wavefiles

PCP-based-model

SPCPS-based-model

GSPCPS-based-model

Wavefiles
PCP-based-model SPCPS-based-model GSPCPS-based-model

Publications
1. Kumud Tripathi, Parakrant Sarkar, and K. Sreenivasa Rao"Sentence Based Discourse Classification for Hindi Story Text-to-Speech (TTS) System", in Thirteenth International Conference on Natural Language Processing (ICON-2016), IIT (BHU), Varanasi, India, 19-20 December, 2016 Link
2. Parakrant Sarkar, and K. Sreenivasa Rao, "Development of Story Text-to-Speech System based on Story Genres", in Workshop on Machine Learning in Speech and Language Processing (MLSLP 2016), Google San Fransisco, USA, 13 September, 2016 Link
3. Parakrant Sarkar, and K. Sreenivasa Rao, "Analysis and Modeling Pauses for Synthesis of Storytelling Speech based on Discourse modes", in Proceedings of the IEEE International Conference on Contemporary Computing (IC3 2015), JIIT Noida, India, 11-13 August, 2015. IEEE-link
4. Parakrant Sarkar, and K. Sreenivasa Rao, "Modeling Pauses for Synthesis of Storytelling Style Speech Using Unsupervised Word Features'', Second International Symposium on Computer Vision and the Internet (VisionNet'15), Procedia Computer Science, Volume 58, Pages 42-49, Kochi, India, August, 2015. IEEE-link
5. Parakrant Sarkar, and K. Sreenivasa Rao, "Data-Driven Pause Prediction for Synthesis of Storytelling Style Speech based on Discourse Modes", in Proceedings of the IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT 2015), IIIT Bangalore, India, 10-11 July, 2015. IEEE-link
6. Parakrant Sarkar, and K. Sreenivasa Rao,"Data-Driven Pause Prediction for Speech Synthesis in Storytelling Style Speech",in Proceedings of the IEEE 21st National Conference on Communication (NCC-2015), IIT Bombay, India, 27 February to 1 March, 2015. IEEE-link
7. Rashmi Verma, Parakrant Sarkar, and K. Sreenivasa Rao, "Conversion of Neutral speech to Storytelling Style speech", in Proceedings of the eighth IEEE International Conference on Advances in Pattern Recognition (ICAPR 2015), ISI Kolkata, India, 04-07 January, 2015. IEEE-link
8. Parakrant Sarkar, Arijul Haque, Arup Kumar Dutta, Gurunath Reddy M., Harikrishna D. M., Prasenjit Dhara, Rashmi Verma, N. P. Narendra, Sunil Kr. S.B., Jainath Yadav, K. Sreenivasa Rao, "Designing prosody rule-set for converting neutral TTS speech to storytelling style speech for Indian languages Bengali, Hindi and Telugu", in Proceedings of the IEEE International Conference on Contemporary Computing (IC3 2014), JIIT Noida, India, 20-24 August, 2014. IEEE-link