Press releases


3DMark update makes it easier to compare Android and iOS

December 7, 2016


3DMark Android benchmark for smartphones and tablets

We released a minor update for our 3DMark Android benchmarking app today.

3DMark Android v1.6.3428 renames some tests to make comparing scores across platforms more intuitive. Benchmark scores are not affected.

The "Sling Shot using ES 3.1" benchmark test is now called "Sling Shot Extreme". Use Sling Shot Extreme to compare flagship Android devices with the latest iPhone and iPad models. The test uses OpenGL ES 3.1 on compatible Android devices. On iOS, it uses Metal.

The "Sling Shot using ES 3.0" benchmark test is now simply called "Sling Shot". Use Sling Shot to compare mainstream Android devices with popular iPhone and iPad models. Sling Shot uses OpenGL ES 3.0 on both Android and iOS devices. 

These new test names match the names we use in the iOS app, making it easier and more natural to compare scores across platforms.

Download

3DMark is a free Android benchmark app available from Google Play.

Download 3DMark Android benchmark from Google Play



A new, easier way to submit your 3DMark scores to HWBOT

November 30, 2016


3DMark benchmark

HWBOT is a popular community for competitive overclocking and benchmark leaderboards. Whether you are new to overclocking or an old hand, HWBOT is the place to go for overclocking contests, rankings, and esports.

Submitting 3DMark scores to HWBOT has traditionally involved manually copying over results, entering hardware details and uploading validation screenshots. For the past few weeks, we've been working with the HWBOT team to make this process easier and faster.

Today, we're launching our new streamlined submission process. Starting with 3DMark Time Spy, you can now submit scores to HWBOT with one click on 3dmark.com. We hope to extend this feature to other tests in the future.

How do I link my HWBOT account to my Futuremark account?

Connecting your HWBOT account is easy. If you don't have an HWBOT account, you can create one here.

  • Sign in to 3dmark.com and go to the Settings page.
  • Scroll down to the Connect HWBOT Account section.
  • Enter your HWBOT username and password and click the Connect button.

How do I submit my 3DMark Time Spy scores to HWBOT?

Please note that you can only submit results that have passed our score validation checks.

  • Open the result page for the 3DMark Time Spy score that you want to submit to HWBOT.
  • In the top section, below the score, click on the Submit result to HWBOT link.
  • Log in to the HWBOT page that opens.
  • You can now add extra details to your result, for example, cooling information and hardware details.
  • When you are finished, click the Submit benchmark result button to submit your score.



Is your PC ready for VR? Find out with VRMark

November 3, 2016


We are excited to announce the public release of VRMark, our new VR benchmark



The performance requirements for VR games are much higher than for typical PC games. So if you're thinking about buying an HTC Vive or an Oculus Rift this holiday, wouldn't it be good to know that your PC is ready for VR?

VRMark comes with two VR benchmark tests, which you can run on your desktop monitor or on a VR headset. At the end of each test, you'll see whether your PC is VR-ready, and if not, how far it falls short. There is also a free-roaming Experience mode that lets you judge the quality of the VR experience with your own eyes.

Orange Room benchmark

VRMark Orange Room Benchmark

The VRMark Orange Room benchmark shows the impressive level of detail that can be achieved on a PC that meets the recommended hardware requirements for the HTC Vive and Oculus Rift. If your PC passes this test, it's ready for the two most popular VR systems available today.

Blue Room benchmark

VRMark Blue Room Benchmark

The VRMark Blue Room benchmark is a more demanding test with a greater level of detail. It is the ideal benchmark for comparing high-end systems with specs above the recommended requirements for the HTC Vive and Oculus Rift. A PC that passes this test will be able to run the latest VR games at the highest settings, and may even be VR-ready for the next generation of VR headsets.

Results and reporting

VRMark Orange Room Benchmark Result Screen

After running a benchmark, you'll see clearly whether your PC is VR-ready or not. To pass, your PC has to meet or exceed the target frame rate without dropping frames. You also get an overall score, which you can use to compare systems.

Hardware monitoring charts show how your PC performed frame-by-frame. There are charts for frame rate, GPU frequency, GPU load, and GPU temperature.

Experience mode

VR headsets use clever techniques to compensate for missed frames. With Experience mode, you can judge the quality of the VR experience with your own eyes. Explore each scene in your own time in VR or on your monitor.

VRMark Experience mode features free movement, spatial audio, and an interactive flashlight for lighting up the details of the scene.

PCMark for Android benchmark for smart phones and tablets

Available now

You can download and buy VRMark from our online store and from Steam

VRMark Basic Edition - free download

  • See if your PC meets the performance requirements for HTC Vive and Oculus Rift.
  • Test your system's VR readiness with the Orange Room benchmark.
  • Explore the Orange Room in Experience mode.

VRMark Advanced Edition - $19.99   $14.99 until November 10

  • Unlock the Blue Room benchmark for high-performance PCs.
  • See detailed results and hardware monitoring charts.
  • Explore both rooms in Experience mode.
  • Make tests more or less demanding with custom settings.

VRMark Professional Edition - contact us for a quote

  • Licensed for business and commercial use.
  • Script and run tests from the Command Line.


New Android benchmark tests available now

October 13, 2016


PCMark for Android benchmark for smart phones and tablets

Today we are releasing a major update for our PCMark for Android benchmarking app. The update adds two new benchmark tests based on real-world activities and an exciting new way to compare the performance of devices across Android OS version updates.

The best way to benchmark smartphones and tablets is to look at the performance of the device as a whole. PCMark for Android uses tests based on everyday activities. By using real applications and native APIs, scores from PCMark for Android better reflect real-world performance and battery life.

Alex Voica, Senior Marketing Specialist at Imagination Technologies, says, "PCMark is actually a solid example of mobile benchmarking done right."

NEW - Work 2.0 performance benchmark and battery life test

PCMark for Android Work 2.0 benchmark and battery life test

See how your device handles common productivity tasks - browsing the web, editing videos, working with documents and data, and editing photos.

PCMark for Android Work 2.0 benchmark test

Work 2.0 is an improved version of the older Work benchmark. It updates the Web Browsing, Writing, and Photo Editing tests and adds two all-new tests for Video Editing and Data Manipulation. You'll find more details in our earlier post and in the PCMark for Android technical guide.

Use the Work 2.0 benchmark to test the performance and battery life of the latest Android devices. And while we now recommend using Work 2.0, the original Work test is still available in the app. Scores from Work 2.0 are not comparable with the older Work test.

NEW - Computer Vision benchmark

PCMark for Android Computer Vision benchmark

Computer Vision is an exciting field with opportunities for a range of innovative apps and services. The PCMark for Android Computer Vision benchmark measures device performance for a range of image recognition tasks using a number of popular open-source libraries.

TensorFlow is an open-source machine learning library developed by Google. The test uses a pre-trained neural network to recognize different objects in a set of pictures.

ZXing, commonly known as Zebra Crossing, is a multi-format barcode image processing library. The test uses ZXing to read a set of barcodes and QR codes.

Tesseract is an open-source optical character recognition library. The test recognizes and extracts English text from a set of images.

You'll find more information for each test in our previous post and in the PCMark for Android technical guide.

PCMark for Android Work 2.0 benchmark test

NEW - Compare performance across OS versions

With this update we are making it easier to see how major Android OS updates affect a device’s performance.

The in-app Best Devices list is the ideal way to compare your device with the latest smartphones and tablets. The update adds the ability to filter the scores by major Android OS version number. For more details, read our post on comparing smartphone performance across Android OS versions.

PCMark for Android comparing smartphone performane across Android OS versions

A benefit of using a benchmark based on real-world tasks is that you can see the areas where the OS update has the biggest impact. There could be big improvements in some types of activities due to new drivers, OS optimisations or changes to APIs. In other areas, performance might stay the same or even degrade slightly.

What’s new in PCMark for Android v2.0.3705

  • Test performance and battery life with the Work 2.0 benchmark.
  • Measure performance with the Computer Vision benchmark.
  • Compare device performance across Android OS versions.

Download

PCMark for Android is a free benchmark app available from Google Play.

Download PCMark for Android benchmark from Google Play



Comparing smartphone performance across Android OS versions

October 12, 2016


PCMark for Android benchmark for smart phones and tablets

PCMark for Android is getting a major update tomorrow. We've already taken a closer look at the new Work 2.0 and Computer Vision benchmarks. In this post, we’ll present another new feature, the ability to compare scores across Android OS versions.

Android OS updates can breathe new life into older devices. New features and interface design changes can give a device a fresh, modern feel. While beneath the surface, driver optimizations and changes to the operating system can improve performance, stability, and battery life.

With this update, we're making it easier to see how Android OS updates affect a device's performance.

PCMark for Android comparing smartphone performance across Android OS versions

LG Nexus 5 and Samsung Galaxy S5 performance across three Android OS generations

In this post, we'll be digging into our database to take a detailed look at how the performance of two of the most popular smartphones has changed across three Android generations.

We looked into the PCMark for Android Work benchmark scores for the LG Nexus 5 and Samsung Galaxy S5 for Android versions 4.x, 5.x and 6.x.

LG Nexus 5 PCMark for Android Work performance by Android OS version

Samsung Galaxy S5 PCMark for Android Work performance by Android OS version

As you can see, while the general trend is for performance to increase with each new OS version, the change varies by device and activity. In fact, for some tests you can see that performance decreased slightly after the update. It's worth remembering that some Android updates focus more on adding new features and functionality than performance.

In these charts, each score is the average of all the benchmarks results received for that model, with obvious outliers excluded. For a popular device, the average comes from thousands of individual benchmark results from users around the world.

For some devices, these individual benchmark scores can vary a lot. People often run benchmarks under less than ideal conditions, for example, with other apps running in the background, with a device that is already running hot or on a device that is poorly configured.

The charts below show the distribution of individual benchmark results for each model. The peak shows the most common score range.

LG Nexus 5 PCMark for Android Work performance distribution by Android OS version

For the LG Nexus 5, you can see that each Android OS update pushes the curve towards higher scores and better performance.

Samsung Galaxy S5 PCMark for Android Work performance distribution by Android OS version

With the Galaxy S5, we see a much larger jump in performance from Android 4.x to 5.x. There is a slight drop in scores moving to Android 6.x, though this is based on limited data since the Android 6.0 update was only rolled out to Samsung S5 devices very recently.

Next, we look at the performance for specific scenarios in the Work benchmark test, such as web browsing, video playback, writing and photo editing.


LG Nexus 5 PCMark for Android Web Browsing performance distribution by Android OS version

LG Nexus 5 PCMark for Android Writing performance distribution by Android OS version

LG Nexus 5 PCMark for Android Video Playback performance distribution by Android OS version

LG Nexus 5 PCMark for Android Photo Editing performance distribution by Android OS version

Samsung Galaxy S5 PCMark for Android Web Browsing performance distribution by Android OS version

Samsung Galaxy S5 PCMark for Android Writing performance distribution by Android OS version

Samsung Galaxy S5 PCMark for Android Video Playback performance distribution by Android OS version

Samsung Galaxy S5 PCMark for Android Photo Editing performance distribution by Android OS version

It's clear from these charts that one should not generalize when talking about the performance benefits of Android OS updates. The effects can vary greatly by device and by activity. 

OS updates can make wide-reaching changes that affect the whole device. People assume that these updates will bring better performance, but we can see that this is not always the case for every activity. It is easy to misinterpret the benefits of an update if you rely on synthetic tests that only measure CPU performance in isolation.

By using a benchmark based on real-world tasks, you can see the areas where an OS update has the biggest impact. For some activities, there might be big improvements as a result of new drivers, OS optimizations or changes to APIs. In other areas, the performance might stay the same or even degrade.

PCMark for Android is the only benchmark app that can give you these insights into a device's real-world performance. And it will get even better tomorrow when we introduce our new Work 2.0 and Computer Vision benchmark tests.

Compare performance across Android OS versions

The in-app Best Devices screen is the ideal way to compare the latest smartphones and tablets. See which devices top the rankings for performance and battery life. This update gives you the option to filter the Best Devices list by a specific Android OS version. By default, the list shows scores from devices with the same Android OS version as your own device.

  1. Swipe left to the Best Devices screen.
  2. Tap the FILTERS button.
  3. Make a selection from the Test dropdown.
  4. Make a selection from the Android OS version dropdown.

PCMark for Android performance comparison by Android OS version in the Best Devices List

It's easy to compare scores across Android OS versions for your own device.

  1. Swipe left to the Best Devices screen.
  2. Tap the FILTERS button.
  3. Use the Search box to find your device or another model you're considering.
  4. Tap on the device to open the Device Details view. Landscape mode works best for this view.
  5. Scroll down, then select a benchmark test and Android version to compare scores.

PCMark for Android comparing smartphone performane across Android OS versions

Use this feature to see how scores have changed with each OS version upgrade, assess the difference in performance, and identify the areas with significant and modest performance changes.

You can choose any model from the Best Devices list and see a side-by-side comparison with your own device. Areas where the device is superior are highlighted green. Areas where it is inferior are shaded red.

Coming soon

PCMark for Android is a free benchmark app available now from Google Play. You'll be able to compare scores across Android versions after the update is released at 10:00 UTC on Thursday, October 13.

Download PCMark for Android benchmark from Google Play



New Android benchmark tests - Computer Vision in focus

October 10, 2016


PCMark for Android benchmark for smart phones and tablets

Our popular smartphone benchmarking app, PCMark for Android, is getting a major update this week. On October 13, we'll add two new benchmark tests and a neat new way to compare devices across Android OS versions. In this post, we'll be focusing on the new Computer Vision benchmark test. (Make sure you check out our post on the new Work 2.0 benchmark as well.)

Computer Vision is an exciting field at the cutting edge of computer science. Recent advances mean that some computer vision techniques are now possible on mobile devices, opening the door for a host of innovative new apps and services.

We couldn't be more proud to introduce the PCMark for Android Computer Vision benchmark for smartphones and tablets. This new test measures the performance of your device using tests based on popular open-source image recognition and machine learning libraries.

PCMark for Android Computer Vision test

Image recognition with TensorFlow

In this test, a pre-trained neural network tries to identify the objects in a set of photos. This technique has many practical uses in mobile applications such as identifying and classifying images in photography apps, tagging people and places in social networking apps, and helping visually impaired people understand the world around them.

The test uses a TensorFlow Inception model that has been trained with the ImageNet database. There's a lot of jargon here, so let's break it down. TensorFlow is an open-source machine learning library developed and supported by Google. Inception is a deep convolutional neural network architecture developed by Google for TensorFlow that is part of the current state of the art for computer image recognition. ImageNet is an academic data set containing thousands of images that is commonly used for training image recognition systems.

The model is loaded into the memory of your device and tested with a set of images. The test measures the time the model takes to classify each image.

Graphical barcode scanning with ZXing

There are dozens of barcode and QR code scanning apps. These helpful apps cut out the need to type long serial numbers and website addresses.

This test uses ZXing, an open-source, multi-format barcode image processing library, to read a set of barcodes and QR codes. To make the test a better representation of real-world use, the codes have been photographed under poor lighting condition and some have simulated tearing damage. The test measures the time taken to recognize each barcode and QR code.

Optical character recognition with Tesseract

Optical character recognition (OCR) is a technique for recognizing and extracting text from images. OCR has many practical and useful applications on mobile devices, from augmented reality translation apps like Google Translate, to document scanners and business card readers.

Tesseract is an open-source optical character recognition library. This test uses a fork of Tesseract Tools for Android called tess-two. The test uses tess-two to recognize and extract English text from a set of images. The images tested have been photographed in poor lighting conditions to simulate the challenges of OCR in the real-world. The test measures the time taken to recognize and extract the text from each image.

Computer Vision benchmark score

PCMark for Android Computer Vision test result

After running the Computer Vision benchmark, you get an overall score and a result for each subtest. You will also see a detailed hardware monitoring chart that shows how your device's CPU clock speed, temperature and battery level changed during the test.

See how your device compares with the latest smartphones and tablets in the in-app Best Devices List. Tap any device to compare it side-by-side with your own device. You can also filter the list by searching for a specific model, brand, CPU, GPU or SoC.

Coming soon

PCMark for Android is a free benchmark app available now from Google Play. The Computer Vision test will be available at 10:00 UTC on Thursday, October 13.

Download PCMark for Android benchmark from Google Play



What's new in the Work 2.0 Android benchmark

October 7, 2016


PCMark for Android benchmark for smart phones and tablets

Next week, on October 13, we will release a major update for our PCMark for Android benchmarking app. The update will add two new benchmark tests and an exciting new way to compare devices across Android OS versions. In this post, we'll be looking at the new and improved Work 2.0 benchmark and battery life test. 

Each new generation of mobile devices is more powerful than the last. With the latest smartphones and tablets we can perform more tasks at home and work without needing a notebook or desktop PC. 

The new Work 2.0 benchmark shows you how well your device handles common productivity tasks. It benchmarks performance and battery life with tests based on real applications using native APIs. Work 2.0 is an improved version of the original Work benchmark. It updates the Web Browsing, Writing, and Photo Editing tests and adds two all-new tests for Video Editing and Data Manipulation.

Video Editing test

PCMark for Android Work 2.0 Video Editing test

Combine a good-quality camera with decent processing power and your smartphone can act as a mobile movie studio. Mobile video editing apps help you trim and reorder clips, apply filters, and tweak the contrast and tone, before sharing your videos online.

The Video Editing test measures your device’s performance when playing, editing and saving video. The first part of the test measures how well your device performs when applying real-time effects to a number of video clips covering a range of common resolutions and frame rates. The second part of the test measures performance when decoding, editing, encoding and muxing a video.

The test uses OpenGL ES 2.0, the native Android MediaCodec API, and ExoPlayer, a Google-developed, open-source media player that improves on the native Android MediaPlayer API.

Data Manipulation test

PCMark for Android Work 2.0 Data Manipulation test

There are now hundreds of apps that help us quantify our personal lives and work performance. We turn to fitness apps to analyse our workouts and monitor our health. We use personal finance apps to track our bank accounts, credit cards, and stock markets.

The new Data Manipulation test measures the time taken to parse data from various file formats, then records the frame rate while interacting with dynamic charts in real-time. Each test uses 10,000 tuples of data to create bar, line and pie charts, which are then animated in response to common gestures such as swipes and zoom.

Web Browsing 2.0 test

Did you know that more people now go online with smartphones than with PCs? Web Browsing 2.0 measures how well your device performs when displaying and interacting with web content. It tests performance when rendering a web page, scrolling, zooming, searching for content, and re-rendering the page after editing and adding an item.

Web Browsing 2.0 improves on the original Web Browsing test by measuring frame rate as well as the time taken to complete the tasks.

Writing 2.0 test

Tablets and smartphones with larger screens let you carry on working even when you're away from the office. The Writing 2.0 test measures how well your device performs when working with documents and text.

The Writing 2.0 test measures performance while adding text and images to a document and then cutting, copying, and pasting text within the document. Writing 2.0 expands on the original test by additionally measuring performance while creating, displaying, encrypting, and decrypting a PDF document.

Photo Editing 2.0 test

"The best camera is the one you have with you," says award-winning photographer, Chase Jarvis. No wonder more photos are taken on smartphones than cameras these days. Photography apps help us crop, fix, and filter our photos before we share them with friends.

The Photo Editing 2.0 test measures how well your device performs when applying various filters and effects to images. It measures the time taken to open, edit and save a set of images. The measurements include the time taken to move graphics data to and from the CPU and GPU, decode from and encode to JPEG format, and to read and write from the device’s storage.

Photo Editing 2.0 improves on the original test by using the latest android.renderscript API in place of the older android.support.v8.renderscript API that was used previously.

Work 2.0 battery life test

For many people, knowing that a phone will last through the day is the deciding factor when choosing a new model. After all, what use is great performance if it means you're always tethered to a charger?

Unfortunately, some manufacturers set battery life expectations too high by quoting figures based on unrealistic use cases. Measuring battery life with the real-world applications and activities in the Work 2.0 benchmark gives a much more realistic view of how long a device will last in everyday use.

"[PCMark is] one of my favorite battery benchmarks because it mixes together various different tasks, stresses different parts of the hardware, and in general gives a very good idea of how battery life will be when the user is doing many things with their smartphone." - Brandon Chester, Smartphone Editor at AnandTech

Work 2.0 benchmark score

PCMark for Android Work 2.0 benchmark results

After running the Work 2.0 benchmark, you get an overall score and results from each of the subtests. Use these scores to compare your device with other smartphones or tablets. Detailed hardware monitoring charts show you how the CPU clock speed, temperature and battery charge level changed during the benchmark run. You can see how CPU scaling and thermal management affect your device's performance and battery life.

After running the Work 2.0 battery life test, you get the battery life result, an overall Work 2.0 performance score, and a chart showing the performance in each loop of the test. The chart makes it easy to see if the performance changed over time due to thermal issues or other factors.

The in-app Best Devices list is the ideal way to compare the performance, popularity and battery life of the latest Android smartphones and tablets. Tap any device to see a side-by-side comparison with your own device, or search for a specific model, brand, CPU, GPU or SoC.

Coming soon

PCMark for Android is a free benchmark app available now from Google Play. The Work 2.0 update will be available at 10:00 UTC on Thursday, October 13.

Download PCMark for Android benchmark from Google Play



New Android benchmark tests coming on October 13

October 5, 2016


PCMark for Android benchmark app

When we released PCMark for Android two years ago, we introduced a fresh approach to benchmarking smartphones and tablets. We focussed on testing the whole device with tests based on everyday activities like browsing websites, watching videos, working with text, and editing photos. 

A modern smartphone is a complex system of interconnected parts. Everyday activities like catching up on social media, writing an email or taking a photo place different demands on the CPU, GPU, memory, and storage. Narrow synthetic tests that only stress a single component in isolation cannot provide a realistic view of these real-world demands. Only by looking at the device as a complete system can you measure its real-world performance.

"[PCMark] tends to test every aspect of a mobile device, unlike microbenchmarks that can often miss aspects of the system that can affect performance." - Joshua Ho, AnandTech Senior Editor

On October 13, we'll release a major update for PCMark for Android. The update will add two new benchmark tests and an exciting new way to compare devices across Android OS versions. 

We'll be sharing more details about these new tests and features over the next few days. For now, here's an introduction to what's coming.

Work 2.0 performance benchmark and battery life test

PCMark for Android Work 2.0 benchmark and battery life test

Use the Work 2.0 benchmark to see how your device handles common productivity tasks  - browsing the web, editing videos, working with documents and data, and editing photos. Work 2.0 measures performance and battery life with tests based on real applications using native APIs.

Work 2.0 is an improved version of the original Work benchmark. It updates the Web Browsing, Writing, and Photo Editing tests and adds two all-new tests for Video Editing and Data Manipulation.

Computer Vision benchmark

PCMark for Android Computer Vision benchmark

Computer Vision is an exciting field with opportunities for a range of innovative mobile apps and services. The new Computer Vision benchmark uses popular open-source libraries to test your device's performance for image recognition, graphical barcode scanning, and optical character recognition tasks.

Compare performance across Android OS versions

The in-app Best Devices list is a great way to compare your device with the latest smartphones and tablets. The update will add the ability to filter the scores by Android OS version number. It's a neat way to see how software updates affect the performance of a device.

Download

PCMark for Android is a free benchmark app available now from Google Play. The update will be available at 10:00 UTC on Thursday, October 13.

Download PCMark for Android benchmark from Google Play



A Closer Look at Asynchronous Compute in 3DMark Time Spy

July 19, 2016


Every Futuremark benchmark is accompanied by a detailed technical guide that explains how the test works, what it measures, and how the scores are calculated. We create these guides to provide transparency to hardware press who use our benchmarks for reviews, to our industry customers, and for technically-minded gamers and overclockers.

The aim of the technical guide is to bridge the gap between code-level implementation that requires expert knowledge and a typical enthusiast's understanding of modern APIs and real-time graphics. But, occasionally, there is enough interest and excitement around a new feature or technique that a more detailed look is called for. One such technique in DirectX 12 is asynchronous compute. 

In this post, we'll explain how command lists, command queues, and specifically, asynchronous compute are used in 3DMark Time Spy.

Before we dive into the technical details, it is worth explaining how we make benchmarks at Futuremark and the steps we take to ensure that our tests are accurate, relevant, and impartial. 

Benchmark development process

As with every 3DMark release, Time Spy was developed with input from industry leaders including AMD, Intel, Microsoft, NVIDIA, and the other members of our Benchmark Development Program (BDP). 

3DMark Time Spy has been in development for nearly two years, and BDP members have been involved from the start. BDP members receive regular builds throughout development and conduct their own review and testing at each stage. They have access to the source code and can suggest improvements and changes to ensure that the implementation is correct. All development takes place in a single source tree, which means anything suggested by a vendor can be immediately reviewed and commented on by the other vendors. Ultimately, each member approves the final benchmark for release to the press and public. 

Of all the 3DMark tests we have created over the years, Time Spy has probably seen the most scrutiny from our partners. Each vendor had staff spend weeks in our office working with our engineers. The daily source code drops we offer have been downloaded hundreds of times. We have around one thousand emails of vendor communication related to Time Spy. Every detail has been debated and discussed at length. 

When there are conflicting views we listen to all parties to hear their opinions and look at the supporting data they provide. We also talk to game developers to hear their views and experiences. When deciding on the implementation, our main guideline is to ask, "Is this the approach a game developer would take?" when considering whether the gain in performance is sufficient for the amount of developer time required.

Benchmark design and principles

In all Futuremark benchmarks we aim for neutrality by ensuring that all hardware is treated equally. Every device runs the same workload using the same code path. This is the only way to produce results that are fair and comparable.

In the past, we have discussed the option of vendor-specific code paths with our development partners, but they are invariably against it. In many cases, an aggressive optimization path would also require altering the work being done, which means the test would no longer provide a common reference point. And with separate paths for each architecture, not only would the outputs not be comparable, but the paths would be obsolete with every new architecture launch. 

3DMark benchmarks use a path that is heavily optimized for all hardware. This path is developed by working with all vendors to ensure that our engine runs as efficiently as possible on all available hardware. Without vendor support and participation this would not be possible, but we are lucky in having active and dedicated development partners. 

Ultimately, 3DMark aims to predict the performance of games in general. To accomplish this, it needs to be able to predict games that are heavily optimized for one vendor, both vendors, and games that are fairly agnostic. 3DMark is not intended to be a measure of the absolute theoretical maximum performance of hardware. 

Overview of Command List usage in DirectX 12

Quoting from MSDN:

"Most modern GPUs contain multiple independent engines that provide specialized functionality. Many have one or more dedicated copy engines, and a compute engine, usually distinct from the 3D engine. Each of these engines can execute commands in parallel with each other. Direct3D 12 provides granular access to the 3D, compute and copy engines, using queues and command lists.

"The following diagram shows a title's CPU threads, each populating one or more of the copy, compute and 3D queues. The 3D queue can drive all three GPU engines, the compute queue can drive the compute and copy engines, and the copy queue simply the copy engine.

"As the different threads populate the queues, there can be no simple guarantee of the order of execution, hence the need for synchronization mechanisms - when the title requires them."

Four threads sending commands to three queues

Command lists and execution

Unlike the Draw/Dispatch calls in DirectX 11 (with immediate context), in DirectX 12, the recording and execution of command lists are decoupled operations. This means that recording can and does happen as soon as it has all available information and there is no thread limitation on it.

For GPU work to happen, command lists are executed on queues, which come in variants of DIRECT (commonly known as graphics), COMPUTE and COPY. Submission of a command list to a queue can happen on any thread. The D3D runtime serializes and orders the lists within a queue.

Once initiated, multiple queues can execute in parallel. But it is entirely up to the driver and the hardware to decide how to execute the command lists - the game or application cannot affect this decision with the DirectX 12 API.

This parallelism is commonly known as ‘asynchronous compute’ when work is done on the COMPUTE queue at the same time as work is being done on the DIRECT queue. 

DIRECT command list

This command list type supports all types of commands in DirectX 12. This includes Draw calls, compute Dispatches and Copies.

COMPUTE command list

This command list type supports compute Dispatch and Copy commands.

DIRECT queue

This queue can be used for executing all types of command lists supported by DirectX 12. 

COMPUTE queue

This queue accepts compute and copy command lists.

COPY command list and queues

This command list and queue type accepts only copy commands and lists respectively.

Please see MSDN for an introduction to the Design Philosophy of Command Queues and Command Lists, and for more information on Executing and Synchronizing Command Lists

Command List usage in 3DMark Time Spy

The Time Spy engine uses two command queues: a DIRECT queue for graphics and compute and a COMPUTE queue for asynchronous compute. This queue structure is used in every instance regardless of the capabilities of the hardware being tested. 

The number of command lists is large as many tasks have their own command lists, (several copies so that frames can be pre-recorded).

The COPY queue is generally used for streaming assets, which is not needed in Time Spy as we load all assets before the benchmark run begins to ensure the test does not gain a dependency on storage or main memory.

Work items in the COMPUTE queue (in order of submission)

  1. Particle simulation
  2. Light culling and tiling
  3. Environment reflections
  4. Horizon based ambient occlusion
  5. Unshadowed surface illumination

Particle simulation

This pass is recorded and executed at the beginning of a frame because it doesn’t depend on the G-buffer. Thus its recording and submission is done in parallel with recording and submission of geometry draws (G-Buffer construction).

Light culling and tiling, Environment reflections, HBAO and Unshadowed surface illumination

These passes are recorded and submitted in parallel with G-Buffer recording and submission, but executed only after the G-Buffer has finished executing and in parallel with shadow map draws execution. This is because they depend on the G-Buffer, but not on the shadow maps.

Simplified DAG of 3DMark Time Spy queue usage

Simplified DAG of 3DMark Time Spy queue usage

  • Grey tasks are CPU tasks.  
  • Green tasks are submissions to the DIRECT queue. 
  • Red tasks are submissions to the COMPUTE queue
  • Yellow tasks are submissions of synchronization points to the queue they’re pointing to. If there are no dependencies between the tasks then they are executed on the CPU in parallel.
  • Each task encapsulates a complex task substructure which is omitted in this simplified graph for clarity.
  • The async_illumination_commands tasks contain: light culling and tiling, environment reflections, HBAO and unshadowed surface illumination.

The significance of synchronization point tasks can be seen by noting that “execute_async_illumination_commands” cannot be executed on the GPU before “execute_gbuffer_commands” is completed, but the submission happens ahead of the execution unless we are CPU bound. So the GPU needs to know that it should wait for one task to complete execution before a dependent task can begin executing, when the execution is split between queues then this operation should be done by the engine otherwise, a RAW hazard occurs.

The dependency between particle simulation and completion of particle illumination in the previous frame is necessary because simulation happens on the COMPUTE queue which is not synchronized by the Present occurring on the DIRECT queue so a WAR hazard occurs.

Task summary

DIRECT queue - G-buffer draws, shadow map draws, shadowed illumination resolve, and post-processing are executed on the direct queue. G-buffer draws, shadow maps and some parts of post-processing are done with graphics shaders, while illumination resolve and the rest of the post-processing is done in compute shaders.

COMPUTE queue - Particle simulation, light culling and tiling, environment reflections, HBAO and unshadowed surface illumination resolve are executed on the compute queue. All tasks in compute queue must be done in compute shaders.

The order of submission can be obtained from the dependency graph. However, it is entirely up to the driver and the hardware to decide when to actually execute the given list as long as it is executed in order in its queue.

How tasks are submitted to the hardware

Next we will use a Microsoft tool called GPUView to see how Time Spy tasks are queued and executed at the hardware level for a variety of graphics cards models. 

In the images captured from GPUView, the queues that exist on the device context side are at the bottom, while the queues in the top section are the hardware queues, which here are referred to as 3D, Compute and Copy. 

The various colors signify different types of packets and from what process they’re coming from: 

  • Cyan-tinted packets are standard work item packets (darker ones are originating on DIRECT queue, while lighter on the COMPUTE queue)
  • Pink packets are waits
  • Reddish are signals (which release the waits)
  • One type of dashed are presents the other is converted waits. 
  • The packets propagate from top to bottom on the device context queues and on the hardware queues.

NVIDIA GeForce GTX 970

GPUView of NVIDIA GeForce GTX 970 running 3DMark Time Spy

This is a screenshot of a trace taken on an NVIDIA GTX 970 viewed with GPUView. It shows when various packets are entering queues and when they leave them. 

In this context, queues are used to hold all types of work items after being serialized by the D3D runtime as described here. It is possible to discern the type of the associated queue in the engine by the work items found in it. 

The DIRECT queue can be identified by the present packets which are enclosed in this screenshot by a yellow ellipse. On the other hand the COMPUTE queue doesn’t contain any present packets, and overall contains significantly less items than the DIRECT queue because it’s packed better due to lower number of required synchronizations.

GPUView of NVIDIA GeForce GTX 970 running 3DMark Time Spy

This is a zoomed-in image in which the propagation can be observed more closely. The black ellipses signify compute load originating on a dedicated compute queue and being serialized by the driver to the hardware 3D queue. The yellow ellipse is to distinguish the device context DIRECT queue by the present packets.

From this image, we can see that although work is submitted in two separate queues, the queues are merged to execute on a single hardware queue.

AMD Radeon Fury

GPUView of AMD Radeon Fury running 3DMark Time Spy

Here is a corresponding trace from an AMD Radeon Fury. Two dedicated queues can clearly be observed on both the hardware and the device context side. The black ellipses are pointing to a specific compute packet, while the yellow ellipse points to a present packet which helps to identify the DIRECT queue.

NVIDIA GeForce GTX 1080

GPUView of NVIDIA GeForce GTX 1080 running 3DMark Time Spy

Above is a corresponding trace from an NVIDIA GTX 1080. As can be seen the general structures resemble those which are found on AMD Radeon Fury, albeit with extra queues that do not originate from the engine and which contain only synchronization items. From this image we can see that the GTX 1080 has an additional compute queue which accepts packets in parallel with the 3D queue.

Running with asynchronous compute disabled in benchmark settings

You can disable the use of asynchronous compute in the Time Spy benchmark workload by using the toggle found on the Time Spy custom run settings page (in 3DMark Advanced and Professional Editions). Running with asynchronous compute disabled in the benchmark forces all work to be done in the DIRECT queue.

Running without asynchronous compute does not have any effect on image quality. When asynchronous compute is disabled in the benchmark, all work items associated with the COMPUTE queue are simply moved to the DIRECT queue. 

In the case where the driver decides to serialize dedicated compute work into the 3D queue, (the game or application cannot affect this decision), it’s up to the driver to place it in the correct order for execution so that visual quality will not be affected.

AMD Radeon Fury

GPUView of AMD Radeon Fury running 3DMark Time Spy

The image above is a trace from an AMD Radeon Fury with asynchronous compute disabled in the benchmark. All work items exist only on one queue and get submitted to one hardware queue. 

From this image we see that work submitted in a single queue will not be automatically distributed on the hardware even when it has an additional compute queue available. Work is only distributed on the hardware when the work is submitted to two separate queues.

NVIDIA GeForce GTX 970

GPUView of NVIDIA GeForce GTX 970 running 3DMark Time Spy

Here is the corresponding trace taken on an NVIDIA GTX 970, with async compute disabled in Time Spy benchmark settings. Again we can see that there is only one queue. 

NVIDIA GeForce GTX 1080

GPUView of NVIDIA GeForce GTX 1080 running 3DMark Time Spy

Finally, here is a corresponding trace taken on an NVIDIA GeForce GTX 1080. As you can see, the extra COMPUTE queue from the previous trace is gone. From this image we see that, as on the Radeon Fury, when we do not create a separate COMPUTE queue then no automatic separation and submission of tasks by the driver will occur.

In all these situations we can see a clear difference when the benchmark forces all work into the same DIRECT queue, or when we populate both DIRECT and COMPUTE queues.

Summary

Asynchronous compute is one of the most interesting new features in DirectX 12. The work that Time Spy places into the COMPUTE queue and the specific implementation of that work is the result of deep co-operation with all BDP members including AMD, Intel, Microsoft, and NVIDIA among others.

The implementation is the same regardless of the underlying hardware. In the benchmark at large, there are no vendor specific optimizations in order to ensure that all hardware performs the same amount of work. This makes benchmark results from all vendors comparable across multiple generations of hardware.  

Whether work placed in the COMPUTE queue is executed in parallel or in serial is ultimately the decision of the underlying driver. In DirectX 12, by placing items into a different queue the application is simply stating that it allows execution to take place in parallel - it is not a requirement, nor is there a method for making such a demand. This is similar to traditional multi-threaded programming for the CPU - by creating threads we allow and are prepared for execution to happen simultaneously. It is up to the OS to decide how it distributes the work.

For benchmarks to be relevant and useful tools, they must be fair, impartial, and unbiased. This is why 3DMark Time Spy, and all other Futuremark benchmarks, are developed with industry-leading hardware and software partners through our Benchmark Development Program using a process that's been government vetted for fairness and neutrality. This process ensures that our benchmarks are accurate, relevant, and impartial.

We take pride in creating the world’s best and most popular benchmarking software. We also hope our openness in providing this detailed description of new DirectX features, and how we utilize them, has been informative and interesting. 

If you have a question about Time Spy, please contact our Director of Engineering, Jani Joki at jani.joki@ul.com, We always aim to be transparent in our implementation and our process, and we're happy to explain any aspect that hasn't been covered here.



3DMark Time Spy - DirectX 12 benchmark test available now

July 14, 2016


3DMark Time Spy, a new DirectX 12 benchmark test, is now available to download and buy from the Futuremark website and from Steam


3DMark Time Spy

Developed with input from AMD, Intel, Microsoft, NVIDIA, and the other members of our Benchmark Development Program, 3DMark Time Spy is one of the first DirectX 12 apps to be built "the right way" from the ground up to fully realize the performance gains that the new API offers. 

DirectX 12, introduced with Windows 10, is a low-level graphics API that reduces processor overhead. With less overhead and better utilization of modern GPU hardware, a DirectX 12 game engine can draw more objects, textures and effects to the screen. How much more? Take a look at the table below that compares Time Spy with Fire Strike, a high-end DirectX 11 test.

Average processing performed per frame in 3DMark Time Spy and 3DMark FIre Strike

With its pure DirectX 12 engine, which supports new API features like asynchronous compute, explicit multi-adapter, and multi-threading, 3DMark Time Spy is the ideal benchmark for testing the DirectX 12 performance of the latest graphics cards.

3DMark - The Gamer's Benchmark

3DMark is a popular benchmarking application used by millions of gamers, hundreds of hardware review sites and many leading technology companies. With its wide range of benchmarks, you can benchmark everything from tablets and laptops to the latest high-end 4K gaming PCs. 

Time Spy is available as a free update for all Windows editions of 3DMark, including 3DMark Basic Edition and the Steam demo.

3DMark Basic Edition / Steam demo - FREE

Benchmark your PC with a range of tests including Time Spy and Fire Strike.

3DMark Advanced Edition - New price from July 23 - $29.99

Since its release in 2013, 3DMark Advanced Edition owners have enjoyed free updates which have added exclusive new tests and features such as Fire Strike Ultra, the API Overhead feature test, and most recently, the new Stress Tests. With the addition of Time Spy, the price of 3DMark Advanced Edition will go up from $24.99 to $29.99 on July 23. 

3DMark Advanced Edition unlocks all benchmarks and additional features including Time Spy Custom run settings and the Time Spy Stress Test. 

  • Unlock all benchmarks and feature tests including Time Spy.
  • Check your PC's stability with 3DMark Stress Tests.
  • Explore your PC's performance limits with custom settings.
  • Get in-depth insights with interactive performance graphs.
  • Automatically save your results offline.

3DMark Time Spy upgrade - $9.99

This is an optional upgrade for people who bought 3DMark Advanced Edition before July 14, 2016. The upgrade unlocks additional features and settings, but it is not required to run Time Spy benchmark tests.

  • Unlock custom settings for Time Spy.
  • Unlock the Time Spy Stress Test.
  • Unlock the option to skip the Time Spy demo.

3DMark Professional Edition - POA

3DMark Professional Edition is licensed for business and commercial use. It offers additional features designed for business users such as command line automation and scripting, extended Stress Test options, Image Quality Tool, private offline results by default, and result export to XML for further analysis and reporting. Contact us for a quote at sales@futuremark.com.

Special launch offers and promotions

Save 60% on 3DMark Advanced Edition until July 23
$9.99, usually $24.99, then $29.99 after July 23

Existing 3DMark owners, save 50% on Time Spy upgrade until July 23
$4.99, usually $9.99



Recent News

Older News

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998