Download Report

Technical
Article
How combinations of sensors can provide for more intelligent notifications in mobile devices
Written by Ernst Haselsteiner
ams AG
On occasion, smartphones can be remarkably dumb. Take this example: an office is peopled by a
number of workers, each quietly concentrating on his or her own tasks. Suddenly, the silence is broken by a loud burst of pop music – the ringtone of the phone of a colleague who happens to be on a
cigarette break. Left on a desk, the phone vibrates, and as it does so it shuffles towards the edge,
forcing a colleague to jump up and move it to a safe place.
It’s irritating for the people in the office, and it breaks their concentration.
On the face of it, it seems odd that a smartphone could behave in such an unsmart way – after all,
smartphones are normally very aware of their relationship to their environment.
In fact, it requires some additional components, and a carefully thought out combination of various
sensors’ outputs, to enable a mobile device to automatically perform appropriate notifications. This
article sketches out a hardware and software framework that could provide this capability while harnessing functions already available in many mobile devices.
A smartphone’s rich array of sensors
A smartphone already contains many sensors. Some may directly support the provision of smart
notifications:
•
a proximity sensor embedded in the display detects when the phone is held to the user’s
ear. This enables the phone to turn off the display and its touch sensitivity.
•
an ambient light sensor enables the system to adjust the brightness of the display in response to the intensity of the ambient light. Advanced ambient light sensors, such as the
TMD3782x Series from ams, combine digital colour sensing with proximity sensing, to enable more sophisticated display management.
•
an accelerometer determines the orientation of the screen, so that the contents may rotate
as appropriate to fit a landscape or portrait configuration.
•
a microphone may sense the volume of ambient noise
As currently deployed, each sensor is assigned to one main function, and is generally not used for
any other. The one exception is the accelerometer, which is also used in gaming apps to support
the player’s user interface.
Page 1 / 7
Technical
Article
Partial awareness of the environment
The array of sensors in a smartphone today provides a partial ability to support smart notification of
smartphone events such as incoming calls, messages and e-mails. For instance, the device’s ambient light sensor and accelerometer can detect whether the smartphone is still and in a bright environment (because of sunlight or artificial light). This means that the device is not on the user’s person (if it were in a pocket, there would be no ambient light incident on the display). This means that
the vibration function can automatically be disabled.
But user notifications can be made far smarter than this, with the addition of a few extra components.
For instance, the ambient light sensor cannot function when the device is lying face down on a desk
or table. In order for the smartphone to see in both directions, it requires two ambient light sensors,
one on the front and one on the back. Now it can use ambient light information no matter what position it is in.
The same applies to proximity detection: a second proximity sensor is required on the back of the
phone. Modules in the TMD277xx Series from ams combine proximity and ambient light sensing, so
only one extra device needs to be added to the back of the phone. If a colour sensor/proximity sensor module is used, this device can actually measure the colour temperature of the ambient light,
allowing it to distinguish between artificial light (indicating the phone is indoors) and sunlight (indicating that it is outdoors).
Smart notifications also require information about the ambient noise levels. This calls for sense inputs from the microphone, combined with digital signal processing, to determine the volume of
sound in the vicinity of the device. Conversion of the amplitude of the microphone’s output to a
noise volume measurement may be implemented in a dedicated sensor interface to avoid the need
to use the computing power of the device’s main applications processor.
The last hardware piece required in the smart notification system is an ambient temperature sensor.
A temperature-sensitive component such as a PT100 resistor can, with signal processing, enable
the mobile device to determine the temperature in the surrounding air. (Its position on the circuitboard should be chosen to minimise the influence of heat generated by the mobile phone itself.)
The absolute accuracy of the measurement is not important: its function is simply to indicate
whether the device is close to the user’s body.
Now these various inputs can be combined to produce a unified view of the environment that the
mobile phone is in at the time when a notification must be provided to the user (see Figure 1). In
other words, the various measurements must all feed into one algorithm which makes a choice from
a number of possible environments, each with its own set of characteristics.
Page 2 / 7
Technical
Article
Fig. 1: a picture of the current environment can be built up from various types of sensor input
How to combine information from different sensors
Referring to the scenario at the start of this article, the volume sensor alone would have indicated
that the device was in a quiet environment; a smart algorithm could then lower the volume of the audible notification. In addition, the information from the front and back proximity sensors in combination would indicate that the phone was almost certainly lying on a desk or other flat surface. The
phone could use this information to make the decision to disable vibration, which is not necessary
when the device is not on the user’s body.
As this example shows, a combination of sensor inputs enables the device to sense its environment
accurately and to make an intelligent decision about the notification style appropriate to that environment.
Page 3 / 7
Technical
Article
Extending this principle of operation, a number of typical scenarios could be defined, and for each
scenario a profile could be configured. This profile would provide an instruction to the mobile device
on the way it should notify the user. For example:
•
the profile ‘in the user’s pocket’ would prompt the device to vibrate
•
‘in a bag, in motion’ would prompt the use of a high-volume alert
•
‘lying on a desk in a quiet room’ would reduce the volume of the alert
The flow chart in Figure 2 outlines a decision tree which would use the sensor combination to distinguish between seven profiles. A series of accelerometer measurements may be used to decide
whether the mobile device is in motion or not. The temperature sensor is used to judge whether the
device is close to the user’s body. The ambient light sensor can distinguish between indoors and
outdoors (a value of ≥2,000 lux indicates that the device is outdoors), or determine whether the device is in a dark environment such as a bag or pocket.
Page 4 / 7
Technical
Article
Fig. 2: decision tree for determining the most appropriate notification profile
User configuration and self-learning capabilities
A smart phone’s capability is such that the decision tree can be made configurable by the user. In
practice, of course, this could create some complexity, and it might be that few users would in practice modify the default decision tree.
But a self-learning algorithm could certainly enhance the user experience. The default state of the
algorithm contains a simple decision tree and a pre-defined set of notification profiles. But these
rules may be automatically modified in response to user inputs and behaviour.
Page 5 / 7
Technical
Article
For instance, if the user tends to switch the phone to vibrate-only in particular scenarios, the device
can automatically add this modification to the standard profile. By doing so, over time the user will
get the impression that the device is anticipating and understanding her behaviour. Of course, this
adaptation must be implemented cautiously to avoid misinterpreting a small number of random
events.
Conclusion
A smartphone is already equipped with a rich array of sensors that are aware of the environment
around the device. But they operate in an isolated fashion today. By combining their measurements,
the phone can be made to sense complete scenarios that a single sensor on its own cannot perceive. Combined with a decision flow implemented in software, the sensor combination in the phone
can actually mimic human behavior, which uses multiple sensor inputs to make smart responses to
the environment we live in.
The result is a more natural, comfortable and enjoyable experience for the user – and for all those
people who share a living or working environment with the user.
Summary
A smartphone contains many sensors that are capable of detecting the environment around the device. These include a proximity sensor, an ambient light sensor, an accelerometer and a microphone.
As currently deployed, each sensor is assigned to one main function, and is generally not used for
any other.
One aspect of a smartphone’s operation – the notification of incoming calls, messages, e-mails and
alerts – is hampered by this discrete use of sensor inputs. Notifications could be ‘smart’ - that is,
appropriate to the operating environment. Today, they are not smart, which is why, for instance, a
phone will ring at full volume and vibrate when lying on a desk in a quiet office.
By combining sensor inputs, a complete view of the operating environment can be synthesised, and
the notification system’s operation modified in response to this environment.
This article outlines a hardware and software framework for implementing smart notifications, drawing on and supplementing the existing sensing capabilities of smartphones to provide an improved
user experience with little change to the device’s basic design.
Page 6 / 7
Technical
Article
For further information
ams AG
Ernst Haselsteiner
Head of Corporate Software
Tel: +43 3136 500 32376
[email protected]
www.ams.com
Page 7 / 7