Meet Cody Feng: Product Testing Lead at Consumer Reports (CR). What most consumers will often see at the forefront of Consumer Reports’ website is a beautiful row of red, yellow and green scoring for products ranging from air fryers to connected home speakers. These numbers provide more transparency and empower consumers to understand the products and services they buy. Behind these simple numbers are dozens of people who spend hours taking apart devices, meticulously evaluating hundreds of questions that range from security and privacy, to usability and connectivity. This interview aims to highlight the rigorous work our testing team does on a daily basis. Let’s dive into the physical (and digital) nuts and bolts of testing at Consumer Reports.
SN: What is your role at Consumer Reports?
CF: I lead a team of technicians who test consumer products with me. We want to know two things: (1) If the product is collecting too much data and whether they’re collecting this data in appropriate ways. We want to see these issues in the product. (2) If the device is vulnerable to known exploits. Internet of Things (IoT) devices are like mini computers. They may not have the same performance — or protection as a computer. We want to manage exploits. If there is a bad actor, how easy is it for them to hack into a device? Our team also designs and structures tests, generates scores for security and privacy informed by the Digital Standard (which is a framework that focuses on privacy, security, governance and ownership).
I tell my mother I test electronics. Everytime she has an issue with a device or sees something is broken at home, she calls me to fix the devices. I’m the person who repairs the device for her — that’s my hobby. I enjoy repairing things. I buy parts from ifixit.com and repair them.
SN: Why does testing matter?
CF: Usually, privacy and security issues will not show on the surface of the product. When companies design the product, sometimes they only aim to implement desired features that allow the product to work. A company might purchase a product from a manufacturer, put a unique brand label on it and sell it on Amazon without checking how good the security is or whether data is shared with any third party. Testing gives us the opportunity to audit the device and figure out if it is doing inappropriate things behind the scenes like sharing data with data brokers or leaving communications unencrypted in a platform or device.
SN: How does testing work on a high level? Let’s use the example of Smart TVs.
CF: For context, our team has done lots of testing focused on measuring privacy and security. This means I get to test things like: TVs, security cameras, wireless cameras, printers, and health applications.
- Due diligence and advising: Once I get an assignment to a product (like Smart TVs), I communicate with a group of experts at Consumer Reports who have been doing TV testing for years.Then, we talk about what concerns they have, any usual findings they may have found before. No one knows TVs better than this team.
- Reference the Digital Standard: Then, we go to the Digital Standard framework to look through and highlight categories (such as viewing data collection and viewing data benefit) that may apply to the Smart TV.
- Create questions, tools, methods: In our workbook (which for us, is an excel sheet filled with the questions, criteria and instructions on how to conduct a test), I determine a list of questions to answer based on a set of criteria and the categories we highlighted in the Digital Standard. Sometimes I have to modify questions depending on the device (e.g. a printer that does not have a sensor or microphone won’t need to assess for any biometrics data). Once we have the questions, I outline the tools and methodologies I need to address them.
- Finalize workbook: Once we have the process, workflow, questions, tools, methods, and indicators of testing, we finalize the instructions in our workbook. At this point, the people who will be testing will have a guide on what we should do in the beginning, middle, and end of testing — when we restart the TVs.
- Lead training sessions: Once we finalize the workbook, we conduct a training session with the people who will be testing the device. Since we may be conducting tests on privacy and security, those procedures are different from “regular-non-smart” TVs procedures. We need certain tools to analyze network traffic, including static and dynamic analyses. The technicians may not have the knowledge so we run this training to show how to run the test and automate as much as possible.
- Conduct testing: We ask the trained technicians to use the protocol to test the smart TVs and gather the data back to the project leader. In this part of the process, the team evaluates the system using the workbook protocols and we highlight any problems or concerns we found during the test. We then do some extra security audits on devices and mobile applications to test for vulnerabilities. If we find a vulnerability, we talk to the manufacturers and push them to make an update or security fix on the device or application.
- Employ quality assurance: The project leader will verify the data to see if there are any mistakes. If there are mistakes, this is the time we will fix the test protocol and collect data again.
- Add weights: We then add weights into each of the questions to generate the final score for each of the devices.
- Generate score: Once we generate the score, we publish the score to the Consumer Reports website.
SN: In your opinion, what are the biggest challenges in the testing process?
CF: There are 3 things that make testing difficult: (1) It’s hard to execute all of the testing processes in general. We conduct “black box” tests — meaning we use public-facing information to conduct our testing. This is much harder than performing white box or gray box testing where we may have access to internal documents or information such as original code or design frameworks. (2) Some companies make very vague statements about how their product works, how they process user’s data and (3)
Once we do find potential fixes or vulnerabilities, it’s hard to find the proper channel to report the findings and convince those companies or manufacturers to make updates.
For example, some companies don’t have a vulnerability disclosure program or a bug bounty program.
- It’s hard to execute all of the testing categories: The more secure the device is, the harder it is for us to inspect it. The device may have security features that disable certain features that prevent us from conducting certain tests. Let’s take encryption for example. Some mobile applications have advanced features to detect if someone is hacking information network communication between your phone and the server. Our test might want to look at what type of data is going out to the server. This is hard to execute because once they implement those advanced features, we may not be able to monitor what information is being sent from user devices to the server.
- Some companies make super vague statements about how their product works: One of the processes in our privacy tests is to review the company’s privacy policy and terms of use. Some companies only offer the privacy policy on their website which is very short and not very helpful for our testing purposes. In those cases, it contains no information regarding data collection, data use and other important privacy indicators, making it hard to evaluate their privacy policies.
- It can be hard to convince companies to make changes even when we find ways to improve the product’s privacy and security. During our testing process, we make disclosures on what we found to the manufacturer and try to help them do better in the future. However, some companies are not willing to make changes. For example, one time we found a vulnerability about how a company handled passwords. The company did not use password hashing to process user passwords, which could put a lot of people at risk. If a bad actor hacked into the server, they may be able to access all of the stored passwords, creating a massive data breach.
We spend a long time having conversations with manufacturers to use password hashing to handle passwords and in the end, sometimes they fix it, making the device safer for consumers to use.
SN: What is the best thing about your job that most people won’t know
CF: I get to test the latest electronic devices! Take TVs for example — I can see every high end TV of every major brand like LG’s OLED Wallpaper TV. I get to better understand what kind of cutting edge technology they’re using, the trends of technology in certain categories for manufacturers, and perhaps what direction they’re taking these devices. By testing them, we can get a good picture of how this technology works and keep learning.
SN: What is one thing you want people to leave this article knowing about testing?
CF: I want people to know that privacy and security is very important to connected devices. Nowadays, more devices are connected to the internet, making them potentially vulnerable to cyber attacks. A lot of people may not want to pay extra money for added security or privacy features in devices.
Companies should not make privacy or security an add-on and should have strong privacy protocols built into the device from the start to make them safer against bad actors. For consumers, we want people to know privacy and security are important and to pay attention to these features and not to just focus on the feature or usability.
If the company knows that consumers care about privacy and security, they will take notice and pay more attention to the demands of consumers when designing devices in the future.
