Human Machine Interface


In the context of HMI in FACTS4WORKERS we are dealing with three main categories that will be applied for the smart factory solution.

First, providing workers with essential information and knowledge at their workplace involves emerging technologies that enable access to data through their mobility, constant availability and standard interfaces.

Second, interaction with those devices is crucial since they should support the worker as digital assistants and therefore require a special focus on usability and user experience.

Finally, the way information is presented to the worker and knowledge is gathered and distributed is a key factor. Augmented Reality is one important tool for this purpose. The goal is to visualize information at the workplace in an innovative way based on emerging technologies and novel interaction concepts.


Solution: Mobile and Wearable Devices

Mobile and wearable devices provide ubiquitous access to information. Although these devices are not very common on the shop-floor today, we will experience an increased deployment in industrial environments in future. They are equipped with interfaces such as NFC, Wi-Fi, Bluetooth, 4G and sensors like cameras, GPS or vital sensors.
Therefore they allow collecting valuable information about the environment they are used in (context) and can share it instantaneously. Real-time monitoring of processes and notifications are possible through these devices. This is the reason why mobile and wearable devices provide obvious benefits on the shop-floor.

Smart Glasses

Whenever information should to be shown to a worker while he needs his hands free to perform a task, tablets and smartphones are not suitable as digital assistants. However, smart glasses offer exactly this possibility. Depending on the type of smart glass, information is either visualized peripheral in the field of view or full screen in front of one or both eyes. Hence these differences allow choosing the appropriate glasses for the according requirements. We’ll use smart glass in the smart factory whenever hands-free is a requirement and the worker needs intermittent information.

Tablet and Smartphones

Mobile devices such as tablets and smart phones are well known from the consumer market. People are familiar with these devices and therefore easily manage to interact with them. Through their relatively large screens and high resolution they are well suited for information visualisation including documents, plans, drawings, photos, videos and 3D content. Moreover, navigation through data via touch interaction is well established and provides high usability if correctly designed. However, industrial applications on mobile devices require a special focus on usability and user experience since requirements derived from conventional HMIs need to translated to these new devices. For the smart factory solution mobile devices will be one core technology. The focus lies on especially rugged devices and protective casings for tablets that are tailored for the special industrial requirements.

Smart Watches

Smart watches fill the niche between mobile devices and smart glasses since these wearables allow nearly hands-free usage but at the same time rely on familiar features such as touch interaction on the (of course smaller than a smartphone’s) screen. However, current models rely on a connected smartphone, but the smartphone only must be in reach and can be stored separately. Future generations will be able to run native stand-alone applications. We’ll equip workers with smart watches on the shop-floor when information should be monitored and according notifications should inform/alert when necessary.

Solution: Novel Interaction Concepts

Applying mobile and wearable devices on the shop-floor requires taking a closer look at the ways we can interact with those devices. If interaction is cumbersome and does not support the worker in his tasks these digital assistants will fail to improve the workplace. Therefore, we consider different interaction concepts and in some case also a combination of them as promising.

Touch and Gestures

Nowadays we interact with mobile devices via touch and gestures. While this kind of interaction is perceived as intuitive and natural, we face several restrictions within industrial environments. For example, protective clothing and gloves might interfere with some of the most common touch tracking technologies. Moreover, dirt, dust and fluids make special demands on these devices. A possible solution is based on considering contactless interaction. This can be achieved by mid-air gesture control, for example. The worker doesn’t have to touch the device anymore but can interact via gestures he performs in 3D space. Different technologies for this purpose have been investigated and some of them are already commercially available. Examples include the Kinect, leap motion or Myo. In future, especially miniaturized 3D sensing cameras that can be embedded in mobile and wearable devices will path the way for gesture interaction. However, for the successful application of gesture control a well-designed concept based on the combination of intuitive and learned gestures will be crucial.

Voice Control

Besides interacting with mobile devices via touch, voice control is one of the most important emerging interaction techniques. Commercialized through applications such as Siri (Apple), Cortona (Microsoft) and OK Google (Android) we already interact via speech with our phones and tablets. Nevertheless this kind of control is not yet common on the shop-floor. However it allows real hands-free control and based on natural speech processing approaches it supports intuitive interaction. Challenges arise through noisy industrial environments which affect the quality and accuracy of voice control. Combined with other interaction techniques voice control will be an important part of the smart factory solution.

Solution: Augmented Reality

Augmented Reality is a technology that enhances the human perception in an intelligent way. Therefore the field of view is overlaid with digital information, either in form of text, images, videos or 3D models that are perspectively matched with the real environment. Moreover, this information can be context-sensitive, for example only content that is important for a task will be shown to a worker. Especially in the smart factory Augmented Reality can cause an additional benefit since it allows visualizing information in a very intuitive way (e.g. directly on the spot at a machine) and naturally communicating complex content (e.g. show a task through a 3D animation instead of describing it).

Technically, Augmented Reality relies on tracking the real environment via the embedded camera of a device. On smartphones and tablets the content is commonly rendered on top of the video stream. With smart glasses a new way of applying Augmented Reality occurred. See-through displays allow directly overlaying the digital content in the wearer’s line-of-sight. However, it strongly depends on the smart glass model if this “true” Augmented Reality is supported and how convincing the result is based on the embedded camera and motion sensors.