There are two big differences.

First, the tests in Automated Testing Kit will test the whole stack on any target you desire i.e. you get real-world feedback that a specific instance is working correctly. That starts with a developer machine, then moves to a staging server and finally ends with the production server.

Second, typically tests in the Kit are customized for the target site. For example, the included login test in ATK assumes a vanilla Drupal installation. However, during implementation the test would be modified for the unique login needs of the site, which might have a different workflow (like Single Sign-On) or a different error display mechanism.

You certainly can but the point of Automated Testing Kit is to give developers a head start. We've thought through many of the issues you will face and include custom functions (such as Drupal hooks) that make targeting screen content easier.

For instance, the tests that include emails even work with virtual mail server Ethereal.email. That way you can verify that site emails are traveling over the public internet. We also have thought through how to send Drush commands to a target server—even when the server is on Pantheon.

You can use these tests to see how we accomplish certain tasks thereby make writing your own tests faster. (Have a better technique than we used? Join the #automated_testing_kit Slack channel and let us know!)

On top of all that, we're always adding new tests.

Yes, but we expect that each team will modify the tests that are provided by Automated Testing Kit and commit them to their own repo. If you have done that, don't re-run the setup script or your existing tests will be overwritten.

A new version will have more tests and more functionality as the tests become more complex. You can upgrade the Drupal module, the helper Javascript libraries and copy the new tests that you want to use.