Friendly - Social Companion App

About the Project

Friendly is an app that utilizes machine learning to help the mentally challenged improve their conversational/social skills both online and in real life. The app consists of three main features, a conversation simulator, a sign language translator, and an anti-spam engine.

Challenges Faced / Learning Points

This project provided me with the opportunity to apply my conceptual understanding of machine learning for the first time. Furthermore, since it was my first time using Kivy, (a mobile centric framework for python) I was learning the ropes while developing the project, which resulted in going through much trial and error. Not only did this project display the importance of persistence, but it also shows the value of being able to learn quickly.

Conversation Simulator

The conversation simulator aims to cultivate socially acceptable behaviors within PWIDs (Persons With Intellectual Disabilties) through remote means where social workers are not needed to be present physically. Furthermore, it acts as a safe space for users to practice their social skills. The conversation simulator provides a conversation prompt for the user to record a response to. The recorded response consist of an audio recording and data from a facial emotion recognition model. The text of the spoken response will then be assessed, with a NLP emotion analysis model being used to find the emotion behind the text. The response will then be grading according to how socially acceptable it is. Finally, feedback is provided to the user so that they can improve on their conversational skills.

Sign Language Translator

From our conversation with social workers at MINDS, we found out that there is a considerate amount of PWIDS who cannot verbally communicate. In addition, they may not always be using the standard sign language that they are taught - Singapore Sign language. Hence this feature aims to teach them the standard sign language as well as allowing them to recognize their own unique sign language. Designed to assist the mentally disabled who are unable to verbally communicate, the sign language translator translates sign language into text. It uses a machine learning model to recognize the signs and has the additional capability of allowing the user to train the model with their own special signs.

Anti-Spam Engine

Many PWIDs may not understand the concept of waiting for a reply before sending another message, and would expect an immediate response. In addition, they can be overzealous when texting someone online, resulting in them starting to spam the recipient when they do not get an immediate reply after sending a message, which may overwhelm or even offend the recipient. This section of the app will prevent that from happening through physically preventing them from being able to spam after detecting that the messaging platform has been opened. Due to the users having to wait for a certain duration of time before being able to type and send messages again, the users are trained in having the patience to wait for a duration of time before sending another message, preferably for the other party to send a reply, instead of mindlessly sending message after message to get an immediate reply. This self discipline is what we want to develop in our users to allow them to navigate the online world without offending anyone, while at the same time hopefully preventing the other party from being offended.