SYSmark2001 White Paper

Introduction

The SYSmark series of benchmarks have been widely used in the industry to study the system performance of desktop machines. With SYSmark 2001, BAPCo takes desktop benchmarking to new levels by introducing a benchmark based on actual Office Productivity and Internet Content Creation scenarios. The scenarios represent snap shots of a day in the life of an Office Productivity application user and Web page developer. Both scenarios incorporate current and forward-looking business usage models that include new application categories. SYSmark 2001 heralds the following features:

• Concurrent execution.
• Response time measurement.
• Running workloads at realistic execution speeds with think time.

These features are explained in the “What’s new in SYSmark 2001” section. BAPCo has also added applications to the benchmark to reflect current usage models for Office Productivity and Internet Content Creation. The new application categories and the applications are:

• Email - Outlook 2000
• File Compression - WinZip 8.0
• Client Database – Access 2000
• Virus Detection – McAfee VirusScan 5.13
• Web Authoring - Macromedia Dreamweaver 4
• Web Animation - Macromedia Flash 5

This document describes the SYSmark 2001 workload model, current usage models, measurement methodology and new features. For detailed instructions on how to install and run SYSmark 2001, please refer to the online documentation provided on the product CD-ROM and the BAPCo web site.

SYSmark 2001 Workload Model

SYSmark 2001 consists of two components:

1. A harness that installs and runs the benchmark and which also displays and stores the results.
2. The set of automation scripts and data that model user interactions with the applications in the scenario. The script and data comprise the “workload”. A brief description of the workloads is given in Table 1. It is divided into two categories that have a total of fourteen applications.

Table 1: Overview of the workloads.
Office Productivity Scenario: This scenario models a corporate user working for an automobile company. The user creates documents using Word, Excel and PowerPoint. The user also accesses email and queries a database. An Internet browser is used to view presentations. The user also invokes a speech to text translation, file compression and virus detection in the background.

Microsoft* Word* 2000	The user opens up an assembly manual (Word) document for a new transmission system. The user makes some formatting changes, inserts step by step diagrams, text additions, applies some different background themes, prints the document, and saves the document in web page format.
Microsoft* Excel* 2000	The user opens up some sales and revenue figures from a spreadsheet. The data is sorted and modified. Various charts related to sales and revenue are created from the data and published in web page format.
Microsoft* PowerPoint* 2000	The user opens up a business presentation to update the previous quarter’s news and sales. Some pictures of automobile manufacturing facilities are inserted into the presentation. Edits and formatting changes are made. Changes are reviewed as they are made using the slide show. The presentation is then given an appropriate background theme and saved in web page format.
Microsoft Access 2000	The user loads last month’s database and cleans up the tables, imports current month’s data from text tables, processes queries, checks results and opens the generated reports and prints them.
Microsoft Outlook 2000	The user searches for text in the messages in the inbox, archives messages, marks all items as read, spell checks, prints and sends some emails.
Netscape* Communicator* 6.0	The user opens an automotive documentation page and looks for a keyword, then checks the source file. After that, the user browses through a PowerPoint presentation saved in web format.
Dragon* NaturallySpeaking* Preferred v.5	The user transcribes a pre-recorded wave file of a document. The transcribing takes place in the background.
WinZip 8.0	The user compresses a collection of video files in the background.
McAfee VirusScan 5.13	The user runs a Virus scan on some files in the background.

Internet Content Creation Scenario: In this scenario, a Web developer creates two web pages for an Extreme Sports company selling a Kayaking product. Images are manipulated in Photoshop and web animations created in Flash are used on a web page created by Dreamweaver. A Kayaking promotional video clip is assembled in Premiere. The page also has links to a video clip that was encoded using Windows Media Encoder.

Adobe* Photoshop* 6.0	The user opens a high definition picture and runs a few sample filters, experiments with the size and orientation, changes pixel/inch ratio, fades the image, adds a border, saves the result image under jpeg format and prints the resultant image.
Adobe* Premiere* 6.0	The user assembles a promotional video for a Kayaking product from stock footage. Various effects are added to make it compelling. The video is then exported in a compressed format.
Macromedia Dreamweaver 4	The user creates two web pages for the online extreme sports company. The first page gives an overview of four kayaking products. The second page gives the details on a specific product. The Flash animation, Photoshop images and the encoded video are imported from the respective applications.
Macromedia Flash 5	The user starts with an FLA file. The following operations are then performed: Step through the desired frames; go to the Library and locate the desired symbol to delete; Import and trace bitmaps; Flip and rotate images; Move, scale and group images. Finally export the movie.
Microsoft Windows Media Encoder 7	The user takes a video clip and encodes it in the background.

Current Usage models

SYSmark 2001 reflects the current business usage models of office productivity users and content creators. SYSmark 2001 emulates current usage models by employing concurrent execution of applications (foreground and background). The user generally has a number of applications running in the background while the focus is on the primary work being done. BAPCo has attempted to model this in the Office Productivity scenario by having file compression, virus scan and a speech to text translation occurring in the background while Office 2000 documents are being created in the foreground. In the Internet Content Creation scenario, video encoding runs concurrently with other web page creation activities.
Today’s desktop office productivity user utilizes more than one application simultaneously to complete day-to-day activities. Many applications remain open on the desktop through out the day. In SYSmark 2001 the Office Productivity scenario has many components of Office 2000 (Word, Excel and PowerPoint) open at the same time. The benchmark runs back and forth between the Office 2000 components. In the Internet Content Creation scenario, the benchmark alternates between Photoshop, Dreamweaver, Flash and Premiere.

Measurement Methodology

The fundamental performance unit in SYSmark 2001 is the “Response Time”. Response time, in the context of SYSmark 2001, is defined as the latency experienced between the submission of a request by the user and the completion of the processing of that request by the application. For example, the response time for a Replace All command in Word 2000 is the time between hitting the Replace All button in the Edit/Replace window and the time that Word 2000 brings up an operation completion window. Each scenario consists of a number of operations performed by a number of applications. Only the response times for each of these operations will be recorded. The time to reposition the mouse or cursor to achieve the operation is not timed, as the user will do this during normal use of the application.
The overall response time for a scenario is the average of all the response times in all the applications that make up that scenario (See Table 1). The average response time for each of the two scenarios is then converted to ratings (explained in the next section – Rating Methodology). The overall SYSmark 2001 rating is derived from the geometric mean of the two scenario ratings.

Rating Methodology

After SYSmark 2001 is run on a system to be evaluated, it assigns the system a performance rating for each scenario (Office Productivity and Internet Content Creation), and an overall rating. The scenario ratings are based on a comparison of average response times between the system being tested and a fixed calibration platform. A rating of 100 indicates the test system has a SYSmark 2001 performance equal to that of the calibration platform using SYSmark 2001, a rating of 200 indicates twice the performance of the calibration platform. The SYSmark 2001 calibration platform has the following configuration:
Motherboard: 815EEA Intel motherboard with P09 Bios
CPU: IntelŪ /PentiumŪ III processor
Core Frequency: 800/133 MHz
Memory: 128 PC 133 CL2 SDRAM
Video/Resolution: Creative Labs GeForce Annihilator 2 with 32 MB DDR onboard memory, 1024x768 resolution, 16 bpp color, 75 Hz monitor resolution.
Video Driver: nVidia v6.31
Disk: 30 GB ATA 100 IBM DPTA-373420.
Operating System: Windows 2000 (Service Pack 1)

What’s new in SYSmark 2001?

SYSmark 2001 has added a number of evolutionary and breakthrough approaches to benchmarking. Here is a list of those additions:
Response Time Metric: Response time, in the context of SYSmark 2001, is defined as the latency experienced between the submission of a request by the user and the completion of the processing of that request by the application. In SYSmark 2001, only the response time of individual operations is included in the performance metric. Unlike SYSmark 2000, which measured end-to-end run time, SYSmark 2001 ignores the time to send keystrokes and mouse clicks to the application. In the real world these operations are done by the human user and so are not timed.
Realistic Benchmark execution Speed with think time: In the real world users type at normal speed with sufficient pauses in between. Running benchmarks as fast as possible using an automation tool which sends super fast keystrokes, is not realistic, as normal users cannot run applications at this speed. In SYSmark 2001, a think time of up to one second is added between operations. The think time is not included in the performance measurement. The think time emulates a desktop users interaction with the operating system and applications. Operating system behavior is more realistic when application interaction has think times (just like a real user) as the OS can devote itself to other book keeping activities (like memory management, scheduling etc). The addition of the think time also makes the run of the automation script more robust on varied platforms.
Background Execution: SYSmark 2001 reflects the current usage models of office productivity users and content creators. The user generally has a number of applications running in the background while the focus is on the primary work being done. BAPCo has attempted to model this in the Office Productivity scenario by having disk compression, virus scan and a speech to text translation occurring in the background while Office 2000 documents are being created in the foreground. In the Internet Content Creation scenario, video encoding runs concurrently with other web page creation activities.
Streamlining Installation/De-installation: SYSmark 2001 installs all the included applications in an encrypted format when the benchmark itself is installed. Subsequent runs of the benchmark only decrypt the installations and invoke the applications. Un-installation of the applications is done only if the user uninstalls SYSmark 2001. This approach adds to the stability of the benchmark by dramatically reducing the number of installs and uninstalls.
New Applications: BAPCo has added Outlook 2000, WinZip 8.0, Access 2000, McAfee VirusScan 5.13, Macromedia Dreamweaver 4 and Macromedia Flash 5 to SYSmark 2001. These new applications reflect the current workflow in today’s desktop scenarios.stalls and uninstalls.
New Applications: BAPCo has added Outlook 2000, WinZip 8.0, Access 2000, McAfee VirusScan 5.13, Macromedia Dreamweaver 4 and Macromedia Flash 5 to SYSmark 2001. These new applications reflect the current workflow in today’s desktop scenarios.