The overall goal of the project is research and development of an innovative methodology for speech therapy assisted by social robots, augmented reality, cloud services and natural language processing, which will support: (1) children and adolescents with communicative disorders in the speech acquisition and practice of language and social skills in an innovative environment and (2) speech therapists in the design of structured game scenarios in the physical, digital and virtual world via an open platform for the synergic integration of assistive technologies (hereinafter the Platform).

Two target groups are defined in the project: children with CD (from preschool and primary  school stage of education) and adolescents with CD (from the middle school stage of education).

The project will achieve the goals set through 6 work packages. Existing methodologies for robot-enhanced speech therapy, augmented reality and NLP will be critically analyzed. New types of socially assistive robots will be researched and analysed. In this way their motor and communicative functionality can be used optimally in creating the methodology through which structured and customised game scenarios will be designed and developed. Mathematical methods for kinematic modelling of robots and 3D design will be used in the development of the new socially assistive robot. Algorithms for design of scenes in virtual and mixed reality will be applied. Via algorithms for use of cloud services and pre-trained transformers that allow machines to understand the complex sub-meaning in sentences in natural language, human-like speech will be generated for verbal communication with the assistive robot. A synergetic integration of the developed assistive methodologies for structured game scenarios enhanced with assistive technologies will be implemented in a common Platform through a stream-based environment in a browser for integration of devices and services. To develop an ergonomic model for using the Platform, we will apply an innovative and appropriate open source technology for intuitive and visual programming through blocks. An experimental approach will be used to validate the functionality of the Platform for support of the speech therapy for children and adolescents with CD who attend the Speech Therapy Centre at South-West University "Neofit Rilski" in Blagoevgrad. The participation of both target groups and speech therapists is considered to be key to success of the experimental research. To perform a quantitative evaluation of non-verbal social signals, existing methods and technologies for eye tracking, head movement, gestures, vocal behaviour, etc. will be innovatively combined.

Scientific research in the development of the Platform will involve repeated cycles of development and testing with the participation of the CACD, ensuring compliance with the ethical rules throughout the project. The platform will be regularly updated and upgraded, following the approach of joint design and inclusion of feedback from all potentially interested parties (speech therapists, psychologists, linguists, phoneticians, children, adolescents, parents, experts from other professional organizations).

When designing the Methodology, an analysis will be made for the development of the currently used assistive technologies in speech therapy, and solutions for integration of more than one AT in an open platform with an ergonomic model for use will be searched. The analysis of new social robots, new technologies for virtual and mixed reality, the potential of visual programming, as well as new methods for the quantitative assessment of non-verbal social cues, will be the key studies that will enable the development of the innovative Methodology proposed in the project. The research method Delphi (a systematic and interactive prediction method) will be applied to the game scenarios so that they can be structured and decomposed into basic skills and behavior for robots and 3D scenes, classified to the specific CD respectively. To recognize the speech therapy context, ATs will use a QR code, which, through keywords (tags), will encode what behavior, 3D scenes or dialogue is expected from the particular assistive technology.

Vector and matrix methods, mathematical apparatus of geometric algebra and screw theory will be used for research, development and control of the movements of the Emo robot which is planned to be included in the project. To research and develop the scenarios assisted by Nao and Furhat robots, an algorithmic approach and a method for visual programming of new skills and behavior will be used through block-based graphic environments - Nao Choregraphe and Furhat Blockly. A kinematic method for programming of the joint movements of the Nao robot will be applied in the Python development environment Naoqi 2.8³⁷. 3D CAD modelling will be used to develop the virtual or mixed reality scenes, as well as the virtual presence of the Nao and Emo robots in the scenes. Unreal Engine (UE)³⁸, which is an open source 3D computer graphics game engine, will be used to develop the scenarios in AR. Written in C++, Unreal Engine is highly portable, supporting a wide range of desktop, mobile, console, and virtual reality platforms. Unreal Engine’s Blueprint visual scripting system allows scripts creation for applications as well as building of blocks such as stand-alone applications.

To integrate the assistive technologies into a common platform, the approach to integrate devices and services in the Internet through a flow-based environment in a Node-RED browser with the possibility to use cloud services will be explored and implemented. By using the concept of creating block-based visual programming languages ​​and editors (VPL), a Blockly⁴⁰ GUI will be developed for the Platform use. For human-like text generation and speech, a client-server communication approach and "on-demand" cloud services which return a text or audio file in response to URL GET or POST requests will be explored. Different language models with generative pre-trained transformer such as Open AI GPT-3⁴¹ and EleutherAI GPT-J⁴² will be investigated and tested. The platform will use the Google Charts Infographics Server to generate QR codes⁴³ which returns an image of the text in response to a URL GET or POST request. To evaluate the behaviour of children and adolescents with CD in real and virtual space, techniques for tracking the movements of the eyes, head, hands, voice, etc. will be applied. Some of the following hardware and software products will be used: 1) Eye tracking in HTC Viva PRO 2) Head and eye tracking in Hololens 2 glasses 3) Tobii Pro eye tracking hardware and software⁴⁴. For observations and objective measurement of certain parameters, both at segmental and at suprasegmental level, a speech analysis approach will be applied via a specialized Computerized Speech Lab hardware and software. A statistical method will be applied to analyze the experimental data through a specialized SPSS Statistics software.