After Apple burst through the four-figure barrier in 2017 we have seen prices for all devices, from iOS and Android manufacturers, grow to the point where the latest S20 is nudging £1,500.
While that for a phone might seem excessive it does include a biometric authentication technology that Fort Knox would be proud of, a camera with the ability to record in 8K and the capacity to reach internet connection speeds the envy of domestic lines. These mobile phones have the capability to carry out tasks we would have needed three different devices to deliver 10 years ago.
One such function was realised at Samsung’s launch of its flagship ‘Galaxy S’ range, earlier this year. At the presentation, it managed to use its latest device’s capability to record video in 8K resolution to stream the show live from San Francisco.
Camera quality has long been a differentiator when it comes to mobile sales, with a growing number of advertising boards featuring the tag line ‘Shot on an iPhone’. Such differentiation is being played out before our eyes with Apple, Samsung, Google and Huawei using a layer of artificial intelligence on top of physical camera sensors.
Speaking at a press event for the S20 series, Samsung mobile head of product management, Paul Scott said, “One of the challenges with the design of a smartphone and the camera is how do you get a zoom lens into a smartphone? With the S20 Ultra we have been able to deliver a super slim design, but still have up to 100 times zoom. It’s an incredible R&D achievement.”
The camera module on the latest device features a 12MP ultra-wide, 108MP wide angle and 48MP telephoto sensor set up with the ability to achieve a 10 times optical zoom. That zoom is achieved using periscopic technology that allows light through the 48MP slot to be beamed back to the sensor. That hardware technology is combined with software that combines multiple pixels together in a process called ‘binning’ once the zoom exceeds 4x.
“People are trying to get the best shot, whether it’s a footballer or it’s a person on stage, and what they always fail on is trying to zoom in and get a good quality shot.
“What you can now do is not only take great photos in low light, but also achieve zoom levels up to 100 times so a user will be able to get the best photo possible.”
The importance of smartphone photography is backed up by HMD Global head of design Raun Forsyth. HMD Global hold the brand licence for Nokia mobile phones and have hit the headlines at Mobile World Congress for its retro handsets but also for the Nokia 9 device which has a five sensor camera set up.
Alongside having the unexpected ability to send trypophobics into a frenzy, the sensors were needed to capture a high dynamic range of brightness and contrast and sense different depths as HMD Global looked to address the needs of professional photographers.
“When we launched the Nokia 9 PureView, featuring the world’s first five camera array with ZEISS Optics, we had photography enthusiasts in mind. They demand the very latest technology to capture their images” said Forsyth.
ARM vice president and general manager of Client Line of business Paul Williamson added that photography is one of the biggest drains on compute performance in mobile devices.
“Improvements in camera systems are driving requirements across the whole system, from the pre-processing directly on the sensor, to post-processing on the main SoC where CPUs and GPUs are heavily used for DSLR-class blurring of pictures to mimic larger optical scopes and Bokeh effect.
“The need for processing more data pushes for higher single thread performance on the CPU, while camera enhancements are pushing ML capabilities across the whole platform, all for the same power budgets and full day or multi-day battery life.”
Growing 5G impact
It’s impossible to talk about mobile technology without mentioning 5G. The latest networks, launched by all four operators, are set to trigger a fourth industrial revolution in manufacturing according to certain analysts, but the immediate focus is on selling phones.
In short, 5G is touted to be the bridge between almost exclusive cellular connectivity for mobile devices, to having the ability to connect all electrical items to a cellular network rather than in house fixed or wireless internet access.
Focus is still very much on the mobile, with downloads speeds, in theory, able to deliver gigabits of data per second speeds - downloading movies and streaming the latest albums is just the start.
Features like edge computing and low latency data request and delivery open up new capabilities including streaming in virtual reality and gaming.
“When you look at the way data is generated and processed, the demands at the edge of the network are increasing significantly more than even in the data centre” suggests David Fraser, Intel technical sales manager for communication service providers.
Above: Periscopic technology, like this example from Oppo, is helping vendors introduce optical zooms on smartphones
“For something like gaming, being able to offer that low latency capability, pushing the content to the network is important so the gamers get that responsiveness and continue to have a good experience without having to process everything on the device.
“We’re seeing [gaming] as one of the key use cases within that 5G space. Especially with the virtual and augmented reality experiences, the latency element becomes even more critical to provide a good experience otherwise the user will feel nauseated.”
On the Network side ZTE president of global sales Xiao Ming said, “there are a lot of the things need to get improved” before those 5G uses can take off.
“There is a whole ecosystem and we, as a key driver, are talking to most of the hardware manufacturers to put effort in; but it is like the chicken and the egg.
“If we don’t push ahead with a 5G ready network, then there will be no possibility for those hardware manufacturers or software businesses to further develop their solutions unless they have that incentive.
“For example, if you want to make VR lenses really popular for people to use, the current shapes won’t fly. It’s too heavy, it’s not easy to wear, the mobility is an issue and it’s really inconsistent with human habits. We’ll need to have more powerful CPUs, GPUs, cameras and VR and AR processing capabilities.”
One of those chickens (or eggs depending on your point of view) is Samsung, which embraced the gaming industry with its Note 9 device launched in summer 2018. “Just from a usage point, one that really stands out for 5G has been gaming,” said Scott. “They get mega competitive in the gaming world and 5G will allow them to connect quickly.”
“When using a device, gamers need the best refresh rate and want the most reactive screen possible. Now we have 120Hz refresh rates so it’s going to look really clear and really smooth.”
Cutting Edge
According to Williamson and Fraser, when it comes to 5G content, we have barely scratched the surface of what will be possible of a mobile device.
While live gaming appears set to be the first use case, Fraser has called for a re-imagination in the way that content is created.
“I think people who are designing applications or games need to look holistically at what compute resource they have on the device, the edge, and back in the data centre, and then design something that spans those three domains.
“The other critical factor is being able to write software that can utilise compute at the edge, the data centre and on the device. It’s becoming more challenging with heterogeneous architectures.”
“5G will open up new use case possibilities and opportunities for developers to take full advantage of a heterogenous platform, to reach maximum performance without draining the battery” said Williamson.
“The availability of 5G and the massive amounts of increased data funnelled through it will push every aspect of the end device, as well as the need for more efficient processing in the data centre.
“5G will unlock a new wave of highly immersive digital experiences which will demand more compute performance.
“There’s a huge opportunity for developers to take advantage of 5G infrastructure to design and deploy these new use cases for edge devices with a drastic reduction in latency and instantaneous responsiveness.”
Unravelling foldables
Another headline grabber has been the foldable device, a number of which were announced last year. Huawei, Samsung and Motorola got in on the act in 2019, although Samsung was the only one to make its Galaxy Fold device available to consumers.
2020 has seen two foldable devices hit the consumer market already with a new form factor. Both Samsung and Motorola decided to go with a horizontal fold with the Z Flip and Razr devices respectively, compared to the vertical fold seen in the Galaxy Fold.
Scott said that, when it comes to the design of devices, Samsung’s priority is “pushing boundaries” whilst also giving practical reasons why someone would go for a folding display. He also said that Samsung has not settled on a specific form factor, adding, “We are absolutely looking at different options.
“The capabilities of Samsung and the investments that we’ve made in screen technology and design are huge. The screen is normally one of the top three influencers in any smartphone decision, along with the camera and battery.
“Consumers are using their devices more than ever before, but they need a screen that is big enough to watch all forms of content and a design that allows our consumers to do that.”
According to Williamson, the chipset manufacturers are ready for the foldable revolution, adding “Foldable devices bring more screens and changing capabilities which require more processing power to build the smarter systems.
“As mobile OEMs are building out these designs, the biggest challenge comes from a software perspective as it’s crucial that apps are able to support all different types of form factors, to provide a seamless user experience across the different screens.
“Mobile processors are powering devices from feature phones, smartphones, and tablets to large-screen devices such as PCs and DTVs.
The load on the system is different, but the capability of the processor is the same and we have seen this proven by the different designs hitting the market from Android, Windows, and Chrome, covering a range of UX from immersive mobile gaming to connected PCs for productivity on the go.
“Multi-tasking and multi-processing are already heavily used today in non-foldable devices. Gaming is a good example where there is a need for efficient multi-core performance on CPUs or GPUs. Foldable designs will increase the need for multi-core performance even further and these new device form factors will bring additional need for peak performance, but also more efficient designs,” Williamson concluded.
