Laravel Envoyer & PHPUnit in Production with a smartly crafted HealthTest

Lucian -

This is a smart mechanism that performs a full deploy, including all composer packages, migrations and everything your deploy needs, and then, if everything is a success, it just switches a symbolic link of the running website to the newly deployed path.

That’s it. It’s a smart hot-swap system that really works.

I find it to be an important piece of the puzzle for any CI pipeline. And this is even smarter when you’re deploying to multiple servers at the same time.

Of  course, this is usually not the right platform to run all your unit, browser, integration or any other tests, especially if it targets a production environment.

However, there is one special test that I use to greatly reduce the chances of shipping a broken increment of the product:

The HealthTest

This is a special kind of PHPUnit test, that makes sure the deploy is healthy enough for actual users to interact with it. Although it can run in the usual testing environment, along with all the other tests, in Envoyer I use it for one purpose: to make sure all the moving parts are correctly set up before activating this release.

The HealthTest contains several quick methods that check the following cases:

  • .env entries that need to be there, especially new ones;
  • Database connections such as MySQL, MariaDB, InfluxDB, MongoDB, CouchDB etc.;
  • Amazon SQS queues, DynamoDB or any AWS service;
  • Writing permissions for certain folders, such as cache;
  • Pretty much everything that’s being influenced by the environment and un-staged files;

Consider the following scenario

Development adds a new Amazon SQS queue that’s being used. All the unit and browser tests are passed, because the ‘testing’ environment is mocking all external access. Rumors say it’s a good practice. The development environment has been set up with real, live queues in order to prove that the code actually works. And some weeks later, it’s all being shipped to production. But wait! Nobody bothered to create the production queues.

Guess what!? The new product increment will fail in production because the production queues do not exist. And, it would have been a mistake to have production and development environments use the same queue. Developers would and should, purge the queue at any time, push items that shouldn’t be processed by production code etc.

The testers and the developer might not even notice this until someone complains that something doesn’t work. Here’s where the HealthTest comes into play. As soon as the code passes all checks (via Travis for example), and it’s considered stable to be shipped, Envoyer deploys the code, along with all the packages and everything, and before activating the release, it makes sure the system has everything configured to run in production. How cool is that?

The health check code

Here’s what our HealthTest.php looks like:

<?php

namespace Tests;

use App\Location;
use Aws\Sqs\SqsClient;
use Illuminate\Support\Collection;
use InfluxDB\Client as InfluxClient;
use InfluxDB\Point;
use InfluxDB\Database;

/**
 * System Health Check Test
 *
 * This test is to be run before each deploy activation.
 * It checks all the moving parts and makes sure the deploy is
 * going to run as expected.
 *
 * Run this without the phpunit.xml configuration, as it also tests
 * that the proper queues, databases, environment, URLs and everything
 * is set up correctly, and running in `testing` environment will cause
 * it to check against a non-working environment.
 *
 * Run with something like:
 *
 * phpunit --no-configuration --color ./tests/HealthTest.php
 *
 * Class HealthTest
 * @package Tests
 */
class HealthTest extends TestCase
{
    /** @test */
    public function valid_application_environment()
    {
        $this->assertContains(
            env('APP_ENV'),
            ['dev', 'test', 'testing', 'production'],
            'APP_ENV not valid: ' . env('APP_ENV')
        );
    }

    /** @test */
    public function valid_environment_key()
    {
        $this->assertNotEmpty(env('APP_KEY'), 'APP_KEY');
    }

    /** @test */
    public function valid_queue_driver()
    {
        if (env('APP_ENV') !== 'testing') {
            $this->assertEquals(
                'sqs',
                env('QUEUE_DRIVER'),
                'SQS Queue driver not selected'
            );
        }
    }

    /** @test */
    public function environment_influxdb_valid_configuration()
    {
        $this->assertNotEmpty(env('INFLUXDB_HOST'), 'INFLUXDB_HOST missing');
        $this->assertNotEmpty(env('INFLUXDB_PORT'), 'INFLUXDB_PORT missing');
        $this->assertNotEmpty(env('INFLUXDB_USER'), 'INFLUXDB_USER missing');
        $this->assertNotEmpty(env('INFLUXDB_PASSWORD'), 'INFLUXDB_PASSWORD missing');
        $this->assertNotNull(env('INFLUXDB_SSL', null), 'INFLUXDB_SSL missing');
    }

    /** @test */
    public function working_influxdb_connection()
    {
        $client = new InfluxClient(
            env('INFLUXDB_HOST'),
            env('INFLUXDB_PORT'),
            env('INFLUXDB_USER'),
            env('INFLUXDB_PASSWORD'),
            env('INFLUXDB_USE_SSL')
        );
        $database = $client->selectDB('defaultdb');
        $result = $database->writePoints(
            $this->getSamplePoints(),
            Database::PRECISION_MICROSECONDS
        );
        $this->assertNotEmpty($result, 'Failed to write InfluxDB test points');
        $result = $database->query('SELECT * FROM test_metric LIMIT 5');
        $points = $result->getPoints();
        $this->assertNotEmpty($points, 'Failed to retrieve InfluxDB test points');
    }

    /** @test */
    public function environment_valid_database_configuration()
    {
        $this->assertNotEmpty(env('DB_CONNECTION'), 'DB_CONNECTION missing');
        $this->assertNotEmpty(env('DB_HOST'), 'DB_HOST missing');
        $this->assertNotEmpty(env('DB_PORT'), 'DB_PORT missing');
        $this->assertNotEmpty(env('DB_DATABASE'), 'DB_DATABASE missing');
        $this->assertNotEmpty(env('DB_USERNAME'), 'DB_USERNAME missing');
        $this->assertNotEmpty(env('DB_PASSWORD'), 'DB_PASSWORD missing');
    }

    /** @test */
    public function working_database_connection()
    {
        $migrations = \DB::table('migrations')->get();
        $this->assertInstanceOf(Collection::class, $migrations);
        $this->assertNotEmpty(
            $migrations,
            '`migrations` table empty, the system will not work!'
        );
    }

    // . . . . . . various other checks . . . . . . . . .

}

It does look like a generic, simple test, with some particularities. Firstly, the code doesn’t use mocks, and it expects for the tests to be run against the production environment. So no adding test data that might show up in user’s accounts, no dropping tables, no migration of anything at all, just making sure all connections are correctly configured and working.

Since your code might use different connections based on the current environment, I’m conditioning some methods only to test if it can run in certain environments (or not in testing at least):

<?php

/** @test */
public function valid_queue_driver()
{
    if (env('APP_ENV') !== 'testing') {
        // do some live checks here
    }
}

I thought about using markTestSkipped() on the tests that only run in production, but knowingly having the whole test suite saying “OK, but incomplete or skipped tests!” didn’t feel green enough to me.

I love it when tests go either green or red. That yellow thing sounds like an anomaly to me and I avoid it. Especially when I know for sure that those tests will never run against the testing environment, in which tests usually run.

Running the HealthTest

To run this test, make sure you’re not using the phpunit.xml configuration file, which probably sets the testing environment, triggers code-coverage computations or other testing-only configuration options that we don’t want here:

phpunit --no-configuration --color ./tests/HealthTest.php

Always make sure the tests are passed when they are run together with all the others, or you’ll break your testing suite when you run them together with the rest of your tests.

Configuring Envoyer to run the HealthTest

To run the HealthCheck before activating the new release, open your Envoyer project, navigate to Deployment Hooks, click the gear icon next to the Activate Release action, and Add Hook in the Before This Action section.

Lumen Considerations

I have tried the same thing on a Lumen 5.4 framework. And it almost works. Since \Lumen\Lumen\Testing\TestCase always sets the environment to ‘testing’, the above coolness won’t work:

<?php

namespace Laravel\Lumen\Testing;

// . . . . . . . . . .

abstract class TestCase extends BaseTestCase
{
    // . . . . . . . . . .

    /**
     * Refresh the application instance.
     *
     * @return void
     */
    protected function refreshApplication()
    {
        putenv('APP_ENV=testing');

        Facade::clearResolvedInstances();

        $this->app = $this->createApplication();
    }

    // . . . . . . . . . .
}

See it there on line 18? Not a really nice thing to do, but it is what it is. For me, at least, it’s unacceptable to hack the core library, so I’ve found a workaround.

The first change updates the HealthTest’s setUp() and tearDown() methods to set the environment to whatever is in the .env file:

<?php

    /**
     * Make sure we load the RIGHT environment
     */
    public function setUp()
    {
        parent::setUp();
        $dotenv = new Dotenv\Dotenv(__DIR__ . '/..');
        $dotenv->overload();
    }

    /**
     * Have the testing environment back to keep from screwing other tests
     */
    public function tearDown()
    {
        putenv('APP_ENV=testing');
        parent::tearDown();
    }

This will overload what TestCase::refreshApplication() does. But now, it isn’t safe to run the HealthTest class along with the rest of your tests, as it will always use the environment in the .env file, and this will mess up the Continuous Integration tests at least. So you’ll want to pull it out of the test suite, by editing the phpunit.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap/app.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false"
         syntaxCheck="false">
    <testsuites>
        <testsuite name="Unit Test Suite">
            <directory suffix="Test.php">./tests/unit</directory>
        </testsuite>
        <testsuite name="Console Commands Test Suite">
            <directory suffix="Test.php">./tests/console</directory>
        </testsuite>
        <testsuite name="API Test Suite">
            <directory suffix="Test.php">./tests/api</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./app</directory>
        </whitelist>
    </filter>
    <php>
        <env name="APP_ENV" value="testing"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="QUEUE_DRIVER" value="sync"/>
    </php>
</phpunit>

Make sure HealthTest.php is not in any of the directories listed in the <directory> tags in your phpunit.xmlfile. Move it outside if you wish. I have it in the tests/ root folder.

Now, whenever you run the full suite of tests, the HealthTest is excluded. When activating the release in production, only run the HealthTest.

You can apply this logic to Laravel as well if you wish, but it’s not mandatory. Laravel works just fine without this particular change.

Conclusion

That’s it! Now you just have to make sure to update the HealthTest whenever a new configuration is added or any of the (usually external) moving parts are changed in any way.

That feeling when you push the code and go out for lunch, resting assured that everything will be alright, is truly priceless!