Q: How many users can I simulate using Visual Studio Team Edition for Software Testers when conducting a Load Test?

A: You may simulate as many virtual users as your hardware allows. Licensing is based per processor and not per virtual user.

Q: What is a Manual Test?

A: A manual test is a test for which there is no automation. Instead, test steps are outlined in a document for the tester to complete. The tester can then report test results and submit bugs as appropriate.

Q: Does Visual Studio Team System support automation testing (e.g., Compuware)?

A: No. However, Visual Studio Team System provides extensibility points out-of-the-box enabling third-parties to develop tools that target Visual Studio Team System.

Q: What is the Test Explorer?

A: Test Explorer is a convenient way to view all of the tests, including manual, unit, and load tests, for a given project.

Q: What is a Test Project?

A: Test projects are standard Visual Studio projects that are specifically created to contain tests. Test projects enable users to draw a clear line between shipping development code vs. code used for quality assurance. Additionally, test projects enable a general manner of organizing tests from a test development standpoint.

Q: Does the test framework manage deploying tests to machines?

A: Yes. Visual Studio 2005 Team System will include a general framework for test deployment that works in both of our supported scenarios: local and remote execution.

Q: Can I use my own tests in the framework and get individual results?

A: Yes. You can do this in two ways:

1) You can wrap an existing test as a Visual Studio Generic Test and get results for it, or

2) You can make your own test type and integrate this into Visual Studio.

Q: How do I migrate my scripts from existing testing tools from third party or Application Center Test?

A: The easiest way to migrate scripts is to re-record them using the browser recorder.

Q: What protocols do you support?

A: We support recording HTTP and HTTPS only; however, other protocols can be tested using Coded WebTests.

Q: How does Visual Studio Team Edition for Software Testers compare/differ from Application Center Test in Visual Studio 2003?

A: The load test features in Team Edition for Software Testers are a completely new product. The features provided far exceed those features provided by ACT.

Q; Can I use these tools to load test Apache/Java based Web applications?

A: Yes, but the tools provided are productivity features designed to ease testing of ASP.NET applications.

Q: Can I test Web services?

A: Yes, although there is no automatic recorder, Web services can be tested by building the SOAP payloads in the test editor.

Q: What is a Web test?

A: A Web test is a test that is used to verify functionality of a Web application. A Web test can be created by recording browser activity and contains a list of Web requests and various properties for each request such as "think time."

Q: What is a load test?

A: A load test is a test that is designed to put a server application under heavy user load to pinpoint performance and/or scalability problems. In Visual Studio, a load test can be based on a Unit Test or a Web Test.

Q: Can I use multiple load clients to generate load?

A: Yes. You can use multiple clients running simultaneously to create very heavy loads.

Q: Can I test ASP and ASP.NET applications?

A: Yes, You can easily test ASP and ASP.NET applications using Visual Studio 2005. There are features for the automatic handling of __VIEWSTATE that make ASP.NET testing easier.

Q: Can I create data driven tests?

A: Yes. Visual Studio 2005 Team System supports the ability to bind input fields and parameters to database files such as Access, SQL Server, or text files.

Q: What protocols are supported?

A: HTTP and HTTPS are supported through the recorder, editors, and test execution engine. Other protocols such as database and other RPC protocols are not directly supported; however, these protocols can be enabled through the extensibility APIs.

Q: Is SSL supported? What about NTLM and Passport?

A: Yes, all of these are supported.

Q: Can I convert my old ACT scripts?

A. There is no conversion utility for ACT scripts or scripts from third party test tool vendors.

Q: Knowing what to monitor during test execution is hard, can you help me?

A; Yes, we provide pre-defined sets of performance counters for common applications such as IIS, ASP.NET, SQL Server and .NET. These pre-defined counter sets can be easily added to a load test.

Q: What information do I receive while executing my tests?

A: During test execution, you receive real-time results that include system resource utilization, request statistics such as response time and content-length, error and overall summary statistics.

How does Testing verify that a product is “Trustworthy”? Is it a fuzzy definition? Is it based on user-perception, marketing, or are there tangible ways we can validate and verify that product is trustworthy ?

Trust means more than secure. A web site or client application could be 100% secure and hacker-proof, yet not trusted by the user. An example might be a bill pay site that does not save your bank account number from session to session. As a user, would you trust that site with your business? Chances are that you wouldn’t, because you would not trust them to pay your bills if they can’t remember your bank account.

Think about the areas and the tools required for building this trust. I am open for discussions

Here is a list of probable "Software Test Automation" interview questions that i have compiled. You can use it for your practice :-)

1. What automating testing tools are you familiar with?
2. How did you use automating testing tools in your job?
3. Describe some problem that you had with automating testing tool.
4. How do you plan test automation?
5. Can test automation improve test effectiveness?
6. What is data - driven automation?
7. What are the main attributes of test automation?
8. Does automation replace manual testing?
9. How will you choose a tool for test automation?
10. How you will evaluate the tool for test automation?
11. What are main benefits of test automation?
12. What could go wrong with test automation?
13. How you will describe testing activities?
14. What testing activities you may want to automate?
15. Describe common problems of test automation.
16. What types of scripting techniques for test automation do you know?
17. What are principles of good testing scripts for automation?
18. What tools are available for support of testing during software development life cycle?
19. Can the activities of test case design be automated?
20. What are the limitations of automating software testing?
21. What skills needed to be a good software test automator?
22. How to find that tools work well with your existing system?
23. Describe some problem that you had with automating testing tool.
24. What are the main attributes of test automation?
25. What testing activities you may want to automate in a project?
26. What are some of the common misconceptions during implementation of an automated testing tools for the first time?

1. Secure by Default

Validate that the product is secure in its default configuration. For this, install the product and configure the product with the default settings. Then perform the appropriate security penetration tests to verify the product is as secure as possible. Afterwards, un-pup the site (either the solution site or the CSharp site) with the default settings, and again perform the security penetration tests to verify the site and its components are as secure as possible. Lastly, configure the SQL roles and user accounts with the security configuration wizard and validate that the product is fully locked down. This step needs to be performed on a single box configuration as well as the secure deployment.

2. Secure by Design

Validate that the product design indeed incorporated security-best practices. Verify that the thread model was adhered to religiously throughout the product life-cycle, and the principle of least privilege and separation of privilege is followed throughout the product.

3. Secure in Deployment

Verify that in deployment, the product can be kept secure, and that the appropriate tools and measures are in place for the customer to diagnose and audit security related issues.

4. Frequency of Security Testing

Perform the security penetration test cases once on a single-box configuration (for both the default configuration and a locked-down configuration) for at least one sprint. For subsequent sprints you can use different configurations

5. Reporting

Threats and vulnerabilities should be logged as bugs. Any security threats that can cause a crash will should be investigated immediately. By most accounts, elevation of privilege is a lot more important than a crash. Anonymous attacks are more important than authenticated ones, and remote attacks trump local ones. So, remote unauthenticated elevation of privilege should be #1 concern. Local authenticated crash is always the lowest on the totem pole. This way you will need to  prioritize the attacks.

Fuzz testing a simple technique for feeding random input to applications. There are two types of fuzz testing categories:

1. Completely random data

2. Partially incorrect data

Random data is a series of arbitrary bytes sent to the interface that is then read by the interface. Partially incorrect data is data that is accurately formed but that might contain invalid values.

Fuzz testing has the following additional characteristics:

1. If the application crashes or hangs, it is considered to fail the test, otherwise it passes. Note that the application does not have to respond in a sensible manner to the input, and it can even quietly exit.

2. Fuzz testing can be automated to a high degree and results can be compared across applications, operating systems, and vendors. There are plenty of tools that exist to perform automated fuzz testing.

Test Cases

1. Perform random data fuzz testing on web service APIs simultaneously. Use a valid body but random data values. Validate that no application crashes. Use of XML fuzzers can help.

2. For the other web services, send boundary condition values (i.e. -1), various Unicode settings, un-escaped and escaped XML entities, type mismatches (i.e. int instead of string), out-of-range data, malformed fragments (i.e. content length says 2K but is really 1K), extraneous headers, binary garbage, extremely large/small payloads, case variations, extra SOAP headers, nonexistent SOAP methods, using too many methods in SOAP method, too few parameters in SOAP methods.

3. Perform code coverage of the input validation code.

Buffer overflows happen when more data is put into a buffer or holding area than the buffer can handle. This is due to a mismatch in processing rates between the producing and consuming processes. This can result in system crashes or the creation of a back door leading to system access. However, attention should be paid to “data-based” buffer overflows, those occurring at the database level due to inconsistencies between the various feature areas.

Test Cases

1. Investigate the design-time and corresponding run-time feature areas for inconsistencies in data length. For instance, create a profile through the runtime and set all the properties to the maximum possible values. Attempt to obtain this profile through the web service, and then save it. Verify there is no data-truncation. Apply this mechanism to all the various feature area dependencies as appropriate.

2. Through the web services, set various properties to binary blobs. Verify there is no data truncation in the database.

3. Pass high integer values where numeric data is expected. Attempt this on 32-bit and 64-bit architecture.

4. On the runtime, create a profile with each property maximized, and numeric data maximized. Attempt this on 32-bit and 64-bit architecture.

5. Due to possibility of integer overflows, pay particular attention to buffer arithmetic done with signed numbers. Unsigned arithmetic provides much better mitigation against attacks.

Denial-of-service attacks, some of the most difficult attacks to protect against, are characterized by an explicit attempt by attackers to prevent legitimate users of a service from using that service. Common denial-of-service attacks include the following:

· Application crash

· CPU starvation

· Memory starvation

· Resource starvation

· Network bandwidth attacks

Out of the categories identified, CPU starvation, memory starvation, resource starvation, and network bandwidth attacks will be tested during the end-to-end stress runs. Thus, we will focus the test cases on application crash denial of service attacks.

Test Cases

1. Perform fuzz testing in an attempt to crash an application component.

2. Identify web service and runtime API that take an input that will cause a server exception. Then repeatedly call the method in a loop, in an attempt to starve the CPU, memory, or a server resource.

3. Generate load by simulating concurrent usage of all the web services, as well as the runtime site. Analyze the server and look for components that allocate large amount of memory. This can be done during the stress testing.

4. Repeatedly refresh the Server caches to look for potential for denial of service threat.

5. In the IIS log files, insert problematic characters, smartly crafted bad input, and really long URL strings. Run the log import DTS task, and look for possible denial of service or information disclosure threats.

6. Send large XML blobs to all web service API in continuous fashion (in a loop or multithreaded environment). Send really small XML blobs to the API in continuous fashion. When testing XML entry points, we can try DTD expansion attacks.

7. In an XML blob, alter the XML so it does an external namespace lookup. Attempt to crash the application leveraging this.

8. In addition, try “algorithm adversary” attacks – feeding data in a way that chokes the algorithm. For instance, binary trees are subject to disordering attacks (making the tree degenerate into a linked list), but balanced trees are not.