Abstract: Silent Speech Interface (SSI) can enable interaction in a new and natural way based on no-audible biosignals generated by the human body. Electroencephalography (EEG) or surface electromyography (sEMG) generated during speech production can be utilized to decode silent speech. However, obtaining complementary information from EEG and sEMG is still challenging. This paper presents a hybrid SSI based on the converter between bimodal physiological electrical signals and audio signals. EEG and sEMG are fused through two sequence-to-sequence models, and multi-task losses are applied to achieve complementarity between speech intention and muscle activity in silent speech. The feasibility of the proposed fusion method is validated in the silent speech dataset, and an average objective character error rate (CER) of 7.22% among eight speakers is obtained. The experimental results show that our bimodal-based hybrid SSI facilitates the conversion of physiological electrical signals to audio.
| Spk id | Text of audio (Mandarin) Text translated to English | |||
| Raw | only sEMG | only EEG | Proposed | |
| 1 | 受伤事故有什么 (shòu shāng shì gù yǒu shén me) What is the injury accident | |||
| 漂流的爱情国语版 (piāo liú de ài qíng guó yǔ bǎn) Drifting Love Mandarin Version | ||||
| 2 | 工人加固塌陷外护栏 (gōng rén jiā gù tā xiàn wài hù lán) Workers reinforce collapsed outer guardrail | |||
| 你对此人的了解 (nǐ duì cǐ rén de liǎo jiě) What you know about this person | ||||
| 3 | 从长久来说没有办法持久 (cóng cháng jiǔ lái shuō méi yǒu bàn fǎ chí jiǔ) There is no way to last in the long run | |||
| 产品也几乎都以别墅为主 (chǎn pǐn yě jī hū dōu yǐ bié shù wéi zhǔ) Almost all products are mainly villas | ||||
| 4 | 二二零四二四七一 (èr èr líng sì èr sì qī yī) Two twenty four two four seven one | |||
| 一会儿看看哦 (yī huì ér kàn kàn o) Check it out later | ||||
| 5 | 目前就到此为止 (mù qián jiù dào cǐ wéi zhǐ) That's all for now | |||
| 广东的作家有什么 (guǎng dōng de zuò jiā yǒu shén me) What do Guangdong writers have | ||||
| 6 | 积极推动体制机制创新 (jī jí tuī dòng tǐ zhì jī zhì chuàng xīn) Actively promote system and mechanism innovation | |||
| 实现社会资源的有效配置 (shí xiàn shè huì zī yuán de yǒu xiào pèi zhì) Realize effective allocation of social resources | ||||
| 7 | 作为该片的出品方 (zuò wéi gāi piàn de chū pǐn fāng) As the producer of the film | |||
| tuì yì yùn dòng yuán yǒu shén me (拼音) Who are the retired athletes | ||||
| 8 | 两人相互依靠欣赏风景 (liǎng rén xiāng hù yī kào xīn shǎng fēng jǐng) Two people rely on each other to enjoy the scenery | |||
| 也能坐拥一套自己的房子 (yě néng zuò yōng yī tào zì jǐ de fáng zi) You can also have a house of your own | ||||
| Spk id | Text of audio (Mandarin) Text translated to English | |||
| Feature Concatenate | Concatenate after Decoder | Add after Encoder | Concatenate after Encoder | |
| 1 | 受伤事故有什么 (shòu shāng shì gù yǒu shén me) What is the injury accident | |||
| 漂流的爱情国语版 (piāo liú de ài qíng guó yǔ bǎn) Drifting Love Mandarin Version | ||||
| 2 | 工人加固塌陷外护栏 (gōng rén jiā gù tā xiàn wài hù lán) Workers reinforce collapsed outer guardrail | |||
| 你对此人的了解 (nǐ duì cǐ rén de liǎo jiě) What you know about this person | ||||
| 3 | 从长久来说没有办法持久 (cóng cháng jiǔ lái shuō méi yǒu bàn fǎ chí jiǔ) There is no way to last in the long run | |||
| 产品也几乎都以别墅为主 (chǎn pǐn yě jī hū dōu yǐ bié shù wéi zhǔ) Almost all products are mainly villas | ||||
| 4 | 二二零四二四七一 (èr èr líng sì èr sì qī yī) Two twenty four two four seven one | |||
| 一会儿看看哦 (yī huì ér kàn kàn o) Check it out later | ||||
| 5 | 目前就到此为止 (mù qián jiù dào cǐ wéi zhǐ) That's all for now | |||
| 广东的作家有什么 (guǎng dōng de zuò jiā yǒu shén me) What do Guangdong writers have | ||||
| 6 | 积极推动体制机制创新 (jī jí tuī dòng tǐ zhì jī zhì chuàng xīn) Actively promote system and mechanism innovation | |||
| 实现社会资源的有效配置 (shí xiàn shè huì zī yuán de yǒu xiào pèi zhì) Realize effective allocation of social resources | ||||
| 7 | 作为该片的出品方 (zuò wéi gāi piàn de chū pǐn fāng) As the producer of the film | |||
| tuì yì yùn dòng yuán yǒu shén me (拼音) Who are the retired athletes | ||||
| 8 | 两人相互依靠欣赏风景 (liǎng rén xiāng hù yī kào xīn shǎng fēng jǐng) Two people rely on each other to enjoy the scenery | |||
| 也能坐拥一套自己的房子 (yě néng zuò yōng yī tào zì jǐ de fáng zi) You can also have a house of your own | ||||