Natural and fully multimodal interaction with the vehicle and its surroundings. BMW Group presents BMW Natural Interaction for the first time at Mobile World Congress 2019.

Munich/Barcelona. Once again, the BMW Group is
revolutionising how drivers interact with their vehicles: At Mobile
World Congress 2019 in Barcelona from 25 – 28 February 2019, the
company will present BMW Natural Interaction. The new system combines
the most advanced voice command technology available with expanded
gesture control and gaze recognition to enable genuine multimodal
operation for the first time. The first BMW Natural Interaction
functions will be available in the BMW iNEXT from 2021.

Just like in interpersonal dialogue, BMW Natural Interaction allows
the driver to use their voice, gestures and gaze at the same time in
various combinations to interact with their vehicle. The preferred
mode of operation can be selected intuitively, according to the
situation and context. Voice commands, gestures and the direction of
gaze can be reliably detected by the vehicle, combined and the desired
operation executed. This free, multimodal interaction is made possible
by speech recognition, optimised sensor technology and
context-sensitive analysis of gestures. Through precise detection of
hand and finger movements, gesture direction – in addition to gesture
type – is also registered for the first time in an extended
interaction space that encompasses the driver’s entire operating
environment. Spoken instructions are registered and processed using
Natural Language Understanding. An intelligent learning algorithm,
which is constantly being refined, combines and interprets the complex
information so that the vehicle can respond accordingly. This creates
a multimodal interactive experience geared towards the driver’s wishes.

By combining different modalities, vehicle functions can be initiated
in different ways. The driver decides how they want to interact, based
on their own personal preferences, habits or the current situation.
So, when the driver is engaged in conversation, they would probably
choose gesture and gaze control; when their eyes are on the road,
better to rely on speech and gestures. In this way, for example, car
windows or the sunroof can be opened or closed, air vents adjusted or
a selection made on the Control Display. If the driver wants to learn
more about vehicle functions, they can also point to buttons and ask
what they do.

With enhanced gesture recognition and the car’s high level of
connectivity, the interaction space is no longer confined to the
interior. For the first time, occupants will be able to interact with
their direct surroundings, such as buildings or parking spaces. Even
complex queries can be answered quickly and easily by pointing a
finger and issuing a voice command. “What’s this building? How long is
that business open? What is this restaurant called? Can I park here
and what does it cost?”

“Customers should be able to communicate with their intelligent
connected vehicle in a totally natural way,” explains Christoph Grote,
Senior Vice President, BMW Group Electronics. “People shouldn’t have
to think about which operating strategy to use to get what they want.
They should always be able to decide freely – and the car should still
understand them. BMW Natural Interaction is also an important step for
the future of autonomous vehicles, when interior concepts will no
longer be geared solely towards the driver’s position and occupants
will have more freedom.”

BMW Natural Interaction: next step for natural operation.

BMW has always played a pioneering role in the
development of systems promoting intuitive operation. In 2001, the BMW
Group became the world’s first car manufacturer to introduce a new
control logic for vehicles, with the iDrive Controller. The
combination of a controller mounted on the centre console with a
multifunctional Control Display replaced a variety of switches,
buttons and indicators – and is still considered a ground-breaking
innovation in the automotive industry. Since 2015, through the use of
a 3D camera, BMW gesture control has enabled easy, non-contact
operation of various vehicle functions. With the launch of the BMW
Operating System 7.0 in 2018, personalised, customisable operation has
reached a whole new level, thanks to fully digital displays, optimised
speech recognition and enhanced gesture control. Depending on their
personal preferences and situation, the driver can choose between the
iDrive controller, steering wheel buttons, touch display or voice and
gesture control. The importance of voice control as the most natural
form of interaction is further underlined by the BMW Intelligent
Personal Assistant. This digital assistant supports the driver in a
variety of situations and learns their routines and habits with every
voice command, making it increasingly easy to operate the vehicle and
access functions and information by voice. As the next logical
development, BMW Natural Interaction also offers the possibility of
using gestures to bring directional and local information into the
interaction for the first time by pointing a finger.

Enhanced sensor technology, precise evaluation using
artificial intelligence.
The advances in recognition
and evaluation of voice commands, gestures and gaze required for
natural driver-vehicle interaction are delivered by improved sensor
and analysis technologies. Using an infrared light signal, the gesture
camera can now capture hand and finger movements in three dimensions
throughout the driver’s entire operating environment and determine a
precise directional vector. For example, pointing a forefinger at the
Control Display and saying a command is sufficient to initiate the
desired operation without touching the screen. The high-definition
camera integrated into the instrument cluster also registers head and
eye direction. The built-in camera technology evaluates the images and
uses them to calculate the required vector data, which is then
processed in the vehicle. To interpret voice instructions quickly and
reliably in addition to gestures, the information transmitted by the
driver to the vehicle in a multimodal manner is combined and evaluated
with the help of artificial intelligence. The algorithm responsible
for interpreting the data in-car is continuously optimised and refined
using machine-learning and evaluation of different operating scenarios.

Beyond the car: environmental interaction through
connectivity.
Thanks to intelligent networking, the area
of BMW Natural Interaction extends beyond the vehicle interior. For
example, the driver can point a finger at objects in their field of
vision and give related voice commands, such as asking for information
about opening hours or customer ratings, or reserving a table at a
restaurant. Thanks to the vehicle’s depth of connectivity, extensive
environmental data and artificial intelligence enable BMW Natural
Interaction to transform the vehicle into a well-informed, helpful
passenger. By connecting digital services, it will be possible to
expand the scope of interaction in the future. For example, when the
driver spots a parking space, they will easily be able to find out
whether they are allowed to park there and what it costs, and then
reserve and pay for it directly without ever pushing a button.

Customer benefits brought to life in mixed-reality experience.

As part of a sophisticated mixed-reality installation, BMW will
immerse visitors to Mobile World Congress 2019 in application
scenarios where they can experience the customer benefits of BMW
Natural Interaction for themselves hands-on. A specially-designed
spatial concept and virtual-reality goggles are used to create a
thoroughly realistic experience that showcases the new possibilities
during a virtual ride in the BMW Vision iNEXT. Visitors discover the
previously unknown freedom of gesture control throughout the area
detected by the gesture camera, which extends across the entire width
of the front vehicle interior. Initially, in training mode,
directional detection of the pointing gesture is visualised by a
dynamic light pulse that follows the direction. Objects the driver can
interact with via pointing are then highlighted. Just how natural this
interaction is becomes apparent in the simple combination of gesture
and language. For example, if the driver points to a side window, this
is visually highlighted with a frame and the voice command
“Open” will then open the chosen window. These totally new
possibilities for interaction with the immediate environment are
revealed during an automated journey through a futuristic city the
driver is unfamiliar with. The vehicle takes over driving and the
visitor embarks on a sightseeing tour of a very different kind –
simply pointing at buildings to obtain all the information they need
about events and exhibitions. Towards the end of the ride, the user
reserves tickets for a cinema they drive by along their route and
streams the trailer for the film directly into the vehicle.

Initial functions in the BMW iNEXT and
outlook.
BMW Natural Interaction paves the way for the
next stage of natural operation in the vehicle and beyond. The free
combination of voice instructions, gestures and gaze creates
multimodal interaction, based on interpersonal communication. The
first BMW Natural Interaction functions will be available in the BMW
iNEXT as early as 2021.

Development of driver-vehicle interaction will make further advances
in parallel. In the future, with the help of artificial intelligence,
the system will continue to learn and enhanced sensor technology will
be able to take occupants’ emotions into account and integrate them
into the interaction in a meaningful fashion. In this way, interaction
between driver and vehicle will become even more personalised and
tailored to the overall situation. Based on experience and depending
on the situation and mood, the intelligent assistant will then be able
to decide whether to wait for instructions or proactively make
suggestions for interaction.

Additional attractions in Barcelona: the BMW Vision iNEXT and
the new BMW 3 Series Sedan with BMW Intelligent Personal Assistant.


Other attractions at the BMW Group stand include the BMW Vision
iNEXT, which provides a glimpse of the future of driving pleasure, and
the new BMW 3 Series Sedan, featuring the functions of the BMW
Intelligent Personal Assistant. As a Vision Vehicle, the BMW Vision
iNEXT showcases totally new ways to experience driving. With the size
and dimensions of a modern Sports Activity Vehicle, it combines
ground-breaking design with the future areas of Automated Driving,
Connectivity, Electrification and Services (D + ACES) set out in the
corporate strategy Number ONE NEXT, and answers the question:
“What will cars look like when they no longer have to be driven by a
person, but still can be?” Among the unique interior features of the
BMW Vision iNEXT is the design principle referred to as “Shy
Tech”, whereby controls are discreetly integrated into interior
surfaces and only become visible when the occupants want to use them.
In the future, it will be possible to operate functions via surfaces
made out of wood or fabric. The materials are awakened with a touch –
for example, to control music playback.
More
information on the BMW Vision iNEXT
.

With a total of more than 15 million vehicles delivered, the BMW 3
Series Sedan is the best-selling premium model worldwide and the core
of the BMW brand. For over 40 years, the BMW 3 Series Sedan has
epitomised sporty driving pleasure in a mid-sized model. With its
dynamic design, agile handling, exemplary efficiency and innovative
functions in the fields of driver assistance, operation and
connectivity, it embodies the typical characteristics of the BMW brand
in concentrated form. More
information on the BMW 3 Series
.

In the new BMW 3 Series, visitors to Mobile World Congress 2019 will
be able to check out the functions of the BMW Intelligent Personal
Assistant available from March 2019 in the brand’s new models. The
intelligent digital companion is familiar with all vehicle functions
and can operate them as needed. The BMW Intelligent Personal Assistant
also learns routines and habits so it can apply them in the
appropriate context in the future. Unique among digital assistants, it
can also be given a name. It not only obeys the driver’s every
command, but is always on hand, even if just for casual conversation.
With every voice command, question and setting, the BMW Intelligent
Personal Assistant gets better and learns more and more preferences
and preferred settings.
More
information on the BMW Intelligent Personal Assistant
.

BMW i Ventures presents portfolio companies in micro-mobility,
AI, digital maps and enterprise security.
BMW i
Ventures is the venture capital arm of the BMW Group, investing in
cutting-edge solutions focused on the company’s current and future
business in the areas of technology, services and the customer
ecosystem. BMW i Ventures operates from offices in Mountain View, San
Francisco and Munich. BMW i Ventures’ four portfolio companies will
present their businesses at the BMW stand at Mobile World Congress
2019 on 26-27 February.
More information on BMW i Ventures.

Lime provides emission-free
micro-mobility in urban areas. Options include pedal bike, e-bike and
electric scooter sharing in +100 markets worldwide, incl. 15 countries
in Europe.
Contact: emeline.chicha@li.me

The GraphcoreIntelligence
Processing Unit (IPU) is specifically designed for machine
intelligence. IPUs let AI researchers undertake entirely new types of
work, not possible using current technologies, to drive the next great
breakthroughs in machine intelligence.
Contact: press@graphcore.ai

Mapillary is a street-level
imagery platform that uses computer vision algorithms to evaluate
street scenes. The result is updated maps enhanced with digital
information that can be used for autonomous systems.
Contact: sandy@mapillary.com

Vera enables businesses to easily
secure and track digital information across all platforms and devices.

Contact: Vera@w2comm.com  

You will find the BMW Group at Mobile World Congress in Hall 5, Stand 5A40.