Open access peer-reviewed chapter

Indian Language Compatible Intelligent User Interfaces

Written By

Sanghamitra Mohanty

Submitted: 16 March 2021 Reviewed: 23 May 2021 Published: 16 June 2021

DOI: 10.5772/intechopen.98525

From the Edited Volume

Software Usability

Edited by Laura M. Castro, David Cabrero and Rüdiger Heimgärtner

Chapter metrics overview

258 Chapter Downloads

View Full Metrics

Abstract

Implicit Human Computer Interaction to explicit context-dependent knowledge processing to meet the requirements of intelligent human being in the present day situation has provoked scientist towards the development of Intelligent User Interfaces to face the society with a demand of smart devices, embedded to the smart environment and dynamic activities. Context-aware situation deals with location and situation focused context like virtual car navigation to context with respect to traffic to climate to the human connected with it immediately and afterwards.

Keywords

  • HCI
  • IUI
  • AI
  • Russel’s Model
  • Plutchik’s Model
  • Table Top Devices
  • command based TTS
  • Speech enabled OCR

1. Introduction

The present day busy and techno savvy life of the people and more with youngsters are provoking people to think of utilizing the brain power more than the other body parts as used in the so to say busy and intelligent life style. Technology is being used by all much more than ever before. It has already been tried with Human Computer Interaction (HCI) applications successfully [1]. The present day presentation, reading, writing and speaking are expected to be done in smart system, where multiple works can be done simultaneously using finger tips, gesture, posture, vision and speech or any other system, which is user friendly and controlled by the brain signals a bit directly giving emphasis to cognition management before execution. Or we can say that in IUI the net impact is the influence of Artificial Intelligence (AI) on Human Computer Interaction on the activities of HCI. Instead of the traditional interfaces one can use Intelligent Interfaces to interact with the day to day life activities starting from study table to any type of action (Figure 1).

Figure 1.

Interdependency of interactions with techniques, machines and interfaces [1].

It is assumed and accepted that Human Computer Interaction has been observed as implicit interaction with respect to traditional communication. Few years back we are habituated to use traditional communication system like telephone, letter writing, personal meeting etc. Later with the development of computers we are in a situation of using the computer, mailing service and mobile phones etc. But as we are entering more towards automation the communication system that we are going to use in near future is totally advanced with respect to the Human Computer Interaction system (HCI) [2, 3, 4]. Automation along with the Artificial Intelligence has influenced our life so much that we are bound to move with the fast moving system in the world (Figure 2).

Figure 2.

The concept diagram for the need and basics of intelligent user interface [2, 3, 4].

It is observed that the availability of automation and speedy life style has forced us to opt for automation. Presently every country is having supportive infrastructure to manage this type of automation. Different type of Interfaces is the basic need to meet the challenges of the smart world through which we are passing through presently. Among the smart Intelligent Interfaces different type of intelligent interfaces may include devices like signal based, gesture based, and speech based but not limited to these types. In a specific way it may be mentioned that IUI uses the following platforms to be effective and useful [5] (Figure 3).

Figure 3.

Components of intelligent user interfaces [5].

The challenges to this device development and utilization are many fold. Among those some very important issues are

  1. Illiteracy

  2. English Illiteracy

  3. Unavailability of Network

  4. Unavailability of Suitable Devices

  5. Language compatibility

and many more human centric issues, which need attention to provide solutions.

Intelligent user interfaces deal with interactive human like interaction to solve any problem. Most effectively without typing can a person deal with the computer and its usage is the challenge of the day.

Advertisement

2. Need of intelligent user interfaces

Explosive materials availability and use in different sectors like national, international and global information extraction and exchange platform. Available present facilities or devices are not enough for futuristic use.

To provide solutions to such situations development of Intelligent User Interfaces are a must. Such Interfaces may include technological concepts like

  1. Image Processing (Gesture)

  2. Speech Processing (Command mode)

  3. Sensors like light, sound, temperature, tactile and pressure sensors to make the Intelligent Interfaces interactive with gesture and posture without writing code or operating with human interventions (tactile sensor for touch sensitive interactions etc.)

  4. Internet of Things (IOT) compatible devices etc.

These factors have motivated the users of different levels to utilize the Multimedia facilities towards the interface development to the challenges due to the complexities in connectivity like telephone network, wireless and satellite communications. Multimedia solutions are a part of the solutions to such problems. Such efforts are supported by the improved microprocessor and storage facilities available now. Such device development may be included in all applications through different WIMP (windows, icons, menus, and pointing) interfaces [6]. It was realized that the real world problem can be better solved by providing Intelligent Interfaces.

Lot of research activities are being carried out and are also presently getting improved due to the dynamic growth of requirements both by human being and machine providing services to mankind in the field of Human Computer Interaction. Techniques embedded with Artificial Intelligence and Machine Learning in the physical devices leads towards the design and development of Intelligent User Interfaces. The Interfaces should be adaptive and adaptable to be intelligent [6, 7].

With respect to implicit Human Computer Interface development in anticipation to the behavior of the user the context expected to be developed the interfaces are to be designed. Most importantly the sensor output based interactions and context dependency can be solved for the purpose using Machine Learning Techniques as well as rule based systems. Thus the sensor based output interpretation and its generic user based modelling to use at large and in particular are challenges for the Intelligent User Interface users as their requirement is dynamic and sources of knowledge in streaming a type of Big Data. It also takes care of emotions of the user and acts accordingly. Context based factors are user based factors like and depends on Physical Environment like location, infrastructure and Human factors like user, social environment. Some of the Interfaces developed by the author in Indian Languages may be discussed here.

Any situation which deals with user and its action and provides output in the form of action which is proactive and functions with prior and appropriate anticipation of the result towards the user need is like motion detector in door opener and locker, beginning of an escalator or conveyor belt, smart communication devices like smart mobiles or tablets which changes it operational conveniences while moving in railway stations, shopping malls long corridors which enables user to pass through being unmindful towards the path. Making on and off of the rooms and room air-conditioners and adjusting the temperature inside the room may be a conference room even is the required setups which needs intelligence interfaces.

It is quite important to find the relationships between the intelligence in a system and the user interaction, which is called the intelligent processing found in the user interface(s) of the system. Such a system should meet the conditions namely effective, natural, or otherwise appropriate interaction of users with the system [8]. Here mainly the system helps the user with pseudo human communication system like speech, gesture or any other form of communication like human beings. Here the interfaces function like an intelligent human being to assist the user to function as per his/her will. This supports the interaction system like human to human interaction with the specially developed system with intelligent interfaces.

Further to say that implicit human computer interaction is an important aspect towards the development of Intelligent Interfaces. The most complex factor which is cognition oriented is computing the human behavior like recognition, interpretation, modelling processing and synthesis of human affects the emotions. Recognized human computer interaction can help in developing effective Intelligent Interfaces. Emotion is of great importance while developing Intelligent User Interfaces as use of those purely depends on the emotions of the users.

Different types of emotions are described by different scientists in due course. Some of those are mentioned below for reference.

Under discrete model Ekman’s Model is well established in treating emotions in terms of discrete and different construct for the analysis of emotion. As per Paul Ekman Model there are seven basic emotions namely Happiness, Sadness, Fear, Disgust, Anger, Contempt and Surprise.

Under Dimensional Model emotions are described mostly through Russel’s Model and Plutchik’s Model.

  1. Russel’s Model by James A Russell describes in two dimensional circular space like arousal and valence with as much as twenty-eight types of emotion having five major classes namely Happy, Relaxed, Sad, Angry and Normal.

  2. Plutchik’s Model by Robert Plutchik [9] has eight standard models described as an extension of Russel’s Model where the eight emotions are anger, anticipation, joy, trust, fear, surprise, sadness and disgust (Figures 4 and 5).

Figure 4.

Russel’s model of emotions, where 28 types of emotions are displayed [10].

Figure 5.

Pultchik’s wheel of emotion showing major eight types of emotion observed [9].

Pultchik’s Wheel of Emotion is described as the wheel of emotion classification and have eight types of resultant emotions. To identify the exact emotion of the user to work accordingly is a sensitive issue.

It is quite important to find the relationships between the intelligence in a system and the user interaction, which is called the intelligent processing is found in the user interface(s) of the system. Such a system should meet the conditions namely effective, natural, or otherwise appropriate interaction of users with the system. Sine here mainly the system helps with human like communication system like speech, gesture or any other form of communication like human beings. Here the interfaces function like an intelligent human being to assist the user to function as per his/her will. This supports the interaction system like human to human interaction with the specially developed system with intelligent interfaces.

Intelligent user Interface systems should have mainly the following criteria like: recommender systems, products, documents, or any other consequential follow-up action. This involves intelligent technology to support information retrieval, appropriate learning environments with learning assistance interface agent with capacity to monitor a situation intelligently assessing the situation and acts for the user of the system using the knowledgebase created with the domain. The main factor of Intelligent Interfaces is the cognition and the agent has to act and react as per the emotions embedded into the need of the user.

Advertisement

3. Challenges in developing intelligent user Interface

The Intelligent Agent that functions behind the IUI has at least the following characteristics [5].

  1. Input analysis

  2. Output functioning

  3. Agent based interaction

  4. Multimodal output presentation interfaces

  5. Well defined model based interfaces

  6. Image based Interaction Interface

  7. Speech based interaction interface

  8. Text analyzing capacity

  9. Heuristic Searching capacity

  10. Intelligent output layout designing capacity

  11. Handling the challenges due to multilingual speech instructions

The performance of an intelligent component may be sensible to failure leading to inappropriate interpretations of commands leading to action and appropriate reaction as Intelligent processing is often relatively difficult for users to predict, understand, and control a demanding situation online. If the demand is bit more elongated depending on relatively extensive information about the users, the intelligent systems may raise certain types of privacy and security risks as private data may not be easy due to security point.

Major challenge to such a situation is evaluating the order of performance of the algorithms used in design and execution of the Intelligent Interfaces. This involves Smart speech processing for talking to the smart human user, Image processing for viewing the activities of the smart human user and Natural Language Processing for understanding the language of a human who is not aware of the language the system is dealing with. As such system will be immediate use of the elderly people and small children. Besides these the factor that influences more is the environment which, controls the functioning of the Intelligent User Interfaces mostly as it changes its features like facilities with respect to user interaction accordingly.

To have Smart Eyes, Ears and Skin (touch) for the IUI are the foremost devices to be developed to enable it to act as the IUI in true sense. Besides the sensor based applications are a basic need for such a system. Presently sensors are operational through Internet. Most of the activities can be controlled by such devices called Internet of Things or IOT. These are the smallest unit in IUI for the largest objective satisfaction. The most important point here to use such devices in IUI is the Security in using IOT as it is internet controlled Cyber intervention are very much expected [11].

Advertisement

4. Different models for the intelligent user interface

User Interfaces are developed using mostly languages like XML, UML and UIML etc.as it deals with different types of users and their requirements in open mode [11].

There are models which are used towards the development of these Intelligent Interfaces.

  1. First one is the Domain Model, which deals with data storage aspects as per the need of the users to complete an assigned work.

  2. Next is the Task Model which is designed to represent the tasks the users may be allowed to perform taking into consideration the time factor of execution. It is mostly ConcurTaskTRees notation based model.

  3. Vanderdonckt and Bodart described the Abstract User Interface Model which includes the user interface expressed in terms of abstract interaction objects [6]. This model is platform independent like Desktop, Laptop, PDA and Smart Phones and modality independent like voice or image input and output.

  4. Then is the Concrete User Interface Model which represents the user interface, which is a composition concrete interaction objects [6]. Here the final user interface is developed to interact with the specified users of the system to support the Human Computer Interaction.

  5. Finally, the Context Model deals with the whole of the context those are to be stored towards the operation and management of the system. It is a combination of the user model, a platform model and an environment model.

  6. This is a combination of Artificial Algorithms to Human Centered Computing or in short Human Centered Artificial Intelligence or HCAI.

Advertisement

5. Challenges for intelligent user interfaces

Success of IUI depends on many factors like Misinterpretation of Cognitive Interaction leading to incorrect understanding by user, erroneous interpretation of instructions by the devices leading to controlling of the system by the user. Depending upon the length of the query demanded by the user the issue of security and privacy will be compromised leading to complex situation [12, 13].

The important factor leading the functioning of the IUI is the dynamism of the user demand depending upon the environment to which the system is exposed to at that time. The solution may be strong algorithm development according to the dynamism of the situation around the user.

One of the most used IUI will be the Tabletop Devices as Intelligent User Interface [14, 15]:

The Tabletop design is an example of IUI to be used in different fields like Education system and Ambient Assistance system as an Augmented Reality facility to cater the needs of the present day situation, when the students aa well as the elderly people are exposed to distance mode of learning and functioning with multimodal facilities around them. This will enhance the growth of interactive tabletop device development activities to cater to the need of academicians and industry people. This has happened due to the increase of interactive systems with digital support brining digital power to table surfaces to support human activities intelligently. To mention the core aspects of interactive tabletop computing devices it may be mentioned that (i) Hardware development of horizontal touch surfaces, (ii) Multimodal Interfacing facilities as per the need like touch, speech, voice and gesture facilities meeting the novel touch and modes of use by the user. A snap shot of the Samsung SUR40 Intelligent User Interface is given below [15] (Figure 6).

Figure 6.

Samsung SUR40 tabletop intelligent user Interface [15].

Advertisement

6. Interface design in Indian language perspectives

One can think of designing an interface intelligent enough to cater the needs of the user. But the category of the users is a big challenge towards the development of the devices. Intelligent User Interface design will mostly depend on the features of Speech Processing, Image Processing and Natural Language Processing Tools.

A screen shot of a Speech Processing Tool developed for Odia Language is given below. This based on the concatenative method of phones used in Odia Language. It helps a user to listen to any text written in Odia language being converted to speech form. It uses the Artificial Intelligence based techniques to cater to the need of the user. This technique has already been tested for many Indian Languages including Indian English. Here the phones, which is the smallest sound unit used in any language are identified and depending upon the pronunciation rules those are concatenated to form a word, thus a sentence. A text may be directly written or read from a file already available like office orders. It reads it out for the user [16, 17, 18, 19] (Figure 7).

Figure 7.

Odia language phone based text-to-speech system [16].

It is observed that many documents are in scanned document form which is not possible to link it to any Text-To-Speech System. Here is a successful attempt to read the content for the needy persons like illiterate and visually challenged [20, 21, 22, 23, 24, 25].

A snap shot of the intelligent user interface for physically challenged is shown below, which works on mouse click as well as speech commands [26, 27, 28, 29, 30] (Figure 8).

Figure 8.

Odia language optical character recognition system which performs on speech commands and reads out the output the OCRed text to the listen [29].

For all such type of Intelligent Interface Development, Natural Language matters as the user may not be English Language Always. The problem to make it more effective towards global use the language factor consisting of Image and Speech are to be handled where the natural process of writing, speaking matters. Thus the factors to be considered are to be considered in connection with the Natural Language Processing aspect.

Here the author has used several user interfaces which are able to manage intelligently the commands or uses expressed in terms of speech or written form. To solve it and exactly interpret to convert it to command for the device different aspects of NLP like Machine Translation, Name Entity Recognition, Grammar restriction free command etc. free like natural speech in command mode are dealt with. Some of the works are mentioned in the papers [31, 32, 33, 34].

Even the gesture is most important as any user will prefer to use gesture to express the mind as at cognition level the brain synthesizes the command and expresses it different form like Speech or Text.

Advertisement

7. Conclusion and immediate future challenges of IUI

While entering to Industrial Revolution 4.0, which is a combination of Automation and Internet, device development is a challenge. More of Automated devices are to be developed, which will use Sensor based commands and will execute it as per the need of the users. Thus IOT based device development is inevitable.

Such work involving the Sensor based includes but not limited to Multiple Sensor based Table Tops. It may be multimodal, multiuser and affordance of the table for multimodal operations accommodating the objects on the Tabletop. Here not only the Tabletops support hand and gesture mode operation but also tangible interaction and often needed visualization [35, 36].

Table Top applications have a massive application in viewing dominated working platform like photo or videos, brain teasing activities like game or competition besides normal computational work to advanced visualization of Dataset dynamically.

The important points for research to be handled for such devices are

  1. Computer Supported Cooperative Work (CSCW), which is the computer-mediated group collaboration and social interactions

  2. Human Computer Interaction (HCI) beyond the desktop setup to individual user actions and reactions.

  3. Ubiquitous Computing (UbiComp) to accommodate variety of input data and output models.

  4. Tangible User Interfaces (TEI) which involves interaction with physical objects on a tabletop and to ovoid those dynamically.

Considering the simple Table top application as one of the simple examples of IUI we observe that the following few parameters are to be taken into consideration while developing the devices. To achieve it mostly we have to go for IOT based services.

The other point to be handled by the scientists are

  1. The Touch Panel, Sensor grid (electrodes) covered by front layer, which enables accurate detection of multi-touch, high positional accuracy.

  2. The other point is Surface/projected capacitive solutions only work with finger or special stylus and not suited for large panels with slower transmission of electrical current. These items are mostly used in Smart Mobiles, which is under metamorphism as per the requirements and demands of the users.

  3. Low-cost manufacturing, Low power consumption compared to other approaches.

  4. Inputs requires pressure on the outer layer.

  5. Reactive to the stylus or when user wearing gloves.

  6. Reduced display quality due to the additional complex layers for execute the functioning details.

  7. Compatible to Mobile Phones, PDAs, Digital Cameras or any other devices under development and use plan.

These are the future challenges which need quick solutions to meet the busy, preoccupied and smart users for their needs. Hope the trend of current research leading to such requirements can provide solutions to these problems as Intelligent User Interfaces are unavoidable in the future generation technology users. It is as dynamic as the need of the end user and the development is a challenge to the developers.

These are leading to the aspects of Augmented Reality and Virtual Reality, which is the Next Generation Computing.

References

  1. 1. “Intelligent User Interfaces: An Introduction”, Morgan Kaufmann, RUIU, San Francisco, 1998, pp. 1-13
  2. 2. “Next Generation Intelligent Interfaces: Implicit Human-Computer Interaction”, Beat Signer, Vrije Universiteit Brussel
  3. 3. “Cross-Media Information Spaces and Architectures”, Beat Signer, Vrije Universiteit Brussel
  4. 4. “Intelligent User Interfaces”, Daniel Sonntag, German Research Centre for Artificial Intelligence (DFKI)
  5. 5. “Intelligent User Interfaces: Past, Present and Future”, V. López Jaquero, et.al., https://www.researchgate.net/publication/226527078
  6. 6. “A Data Processing System for Air Defense”, R. Everett, C. A. Zraket, and H. Bennington, SAGE: , 5(4), 1983
  7. 7. “Intelligent user interfaces: an introduction”, Maybury M. In Proceedings of the 4th international Conference on intelligent User interfaces, ‘IUI’ 99. ACM Press, New York, 1999
  8. 8. “Model-Based Design and Evaluation of Interactive Applications”, F. Paternò, Springer. 1999
  9. 9. “Summarizing Emotions from Text Using Plutchik’s Wheel of Emotions”, Advances in Intelligent Systems Research, volume 166, 7th.M. M. Abasi and Anatoly Beltiukov, Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2019)
  10. 10. “A Circumplex Model of Affect J.A. Russel,, Journal of Personality and Social Psychology, 39(6), 1980, https://content.apa.org/doi/10.1037/h0077714
  11. 11. “Encapsulating Knowledge for Intelligent Automatic Interaction Objects Selection”, Vanderdonckt, J., Bodart, F. In ACM Proc. of the Conf. On Human Factors in Computing Systems INTERCHI’93 (Amsterdam, 24-29 April 1993), S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel & T. White (Eds.), ACM Press, New York, 1993
  12. 12. “Steps to take before Intelligent User Interfaces become real”, Kristina Hook, http://www.sics.se/~kia/, 2000
  13. 13. “Comprehensive Understanding of Intelligent User Interfaces”, S. Shaikh, N. A. Khan and F. B. Solangi, (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 8, No. 6, 2017
  14. 14. “Next Generation User Interfaces Interactive Tabletops and Surfaces”, Beat Signer, Vrije Universiteit Brussel
  15. 15. “Interactive Tabletops and Surfaces - Lecture 07 – Next Generation User Interfaces (4018166FNR)”, 2017
  16. 16. “Syllable based Indian language text to speech system”, S. Mohanty, International Journal of Advances in Engineering & Technology, Vol. 1(2), 2011
  17. 17. “Use of SVM Classifier & MFCC in Speech Emotion Recognition System”, Khyamamayee Dash and Prof. Sanghamitra Mohanty Bhoomika Panda, Debananda Padhi, IJARCSSE, 2012
  18. 18. “Speaker identification using mel frequency cepstralcoefficient and bpnn”, Kshamamayee Dash, Debananda Padhi, Bhoomika Panda, Sanghamitra Mohanty, 2012
  19. 19. “Phonotactic model for spoken language identification in Indian language perspective”, S. Mohanty, International Journal of Computer Applications, 2011
  20. 20. “Speaker Identification using SVM during Oriya Speech Recognition.”, Sanghamitra Mohanty, Basanta Kumar Swain, International Journal of Image, Graphics & Signal Processing, 2015
  21. 21. “Language identification using support vector machine”, Sanghamitra Mohanty, Basanta Kumar Swain, OCOCOSDA, 2010
  22. 22. “Emotion recognition using fuzzy K-means from Oriya speech”, Sanghamitra Mohanty, Basanta Kumar Swain, 2010 for International Conference [ACCTA-2010]
  23. 23. “Language Identification-A Supportive Tool for Multilingual ASR in Indian Perspective”, Sanghamitra Mohanty and Basanta Kumar Swain, ICACIE2019, 2019
  24. 24. “Isolated Word based Spoken Dialogue System using Odia Phones”, Sanghamitra Mohanty and Basanta Kumar Swain, ICICT2019, 2019
  25. 25. “Isolated Word-based Spoken Dialogue System using Odia Phones”, Sanghamitra Mohanty, Basanta Kumar Swain and Chiranji Lal Chowdhary, Recent Advances in Computer Science and Communications, Volume 14, Issue 6, 2021
  26. 26. “Pattern recognition in alphabets of Oriya language using Kohonen neural network”, S. Mohanty, International journal of pattern recognition and artificial intelligence, World Scientific Publication 1997
  27. 27. “An efficient bilingual optical character recognition (English-Oriya) system for printed documents”, S. Mohanty, H. N. Dasbebartta and T.K. Behera, Seventh International Conference on Advances in Pattern Recognition, IEEE Publications, 2009
  28. 28. “Oriya Character recognition using neural networks”, Soumya Mishra, Debashish Nanda, Sanghamitra Mohanty, Special issue of IJCCT, 2010
  29. 29. “A complete OCR development system for Oriya script”, S. Mohanty and H. K. Behera, SIMPLE, 2004
  30. 30. “Algorithm for segmenting script-dependent portion in a bilingual Optical Character Recognition system”, Sanghamitra Mohanty and Himadri Nandini Das Bebartta, Pattern Recognition and Image Analysis, 27, 2017
  31. 31. “Semantic based text classification using wordnets: Indian language perspective”, S Mohanty, PK Santi, Ranjeeta Mishra, RN Mohapatra, Sabyasachi Swain, Proceedings of the 3th International Global WordNejuh t Conf., South Jeju Island, Korea, 2006
  32. 32. “A two stage language independent named entity recognition for indian languages”, S Biswas, MK Mishra, S Acharya Sitanath_biswas, S Mohanty, International Journal of Computer Science and Information Technologies, 2010
  33. 33. “A hybrid Oriya Named Entity Recognition System: harnessing the power of rule”, Sitanath Biswas, SP Mishra, S Acharya, S Mohanty, International Journal of Artificial Intelligence and Expert Systems (IJAE), 2010
  34. 34. “Oriya Language Text Mining Using C5. 0 Algorithm”, Sohag Sundar Nanda, Soumya Mishra, Sanghamitra Mohanty, (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 2 (1) , 2014
  35. 35. “Architecting next-generation user interfaces”, Daniel Wigdor, Proceedings of the International Conference on Advanced Visual Interfaces, AVI 2010, Roma, Italy, 2010
  36. 36. “What’s Hot in Intelligent User Interface”, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16)”, Shimei Pan, Oliver Brdiczka, Giuseppe Carenini and Duen Horng Chau, 2016

Written By

Sanghamitra Mohanty

Submitted: 16 March 2021 Reviewed: 23 May 2021 Published: 16 June 2021