Today I’ve been working on a library I’m building for making it easier to build RESTful API’s with Laravel. It uses an abstract RESTful controller, which inherits from the default Laravel controller, and I wanted to verify that the instantiated controller includes all the traits from the base controller.

However, there was a problem. The only practical way to verify that a class includes a trait is with the class_uses() function, but this doesn’t work if the class inherits from a parent that includes these traits. As the class is abstract, it can’t be instantiated directly, so you must either create a dummy class just for testing that extends it, or mock the class, and that means that class_uses() won’t work. As a result, I needed to first get the parent class, then call class_uses() on that, which is possible, but a bit verbose to do repeatedly for several tests.

Fortunately it’s quite easy to create your own custom assertions in PHPUnit. I started out by setting up the test with the assertion I wanted to have:

<?php
namespace Tests\Unit\Http\Controllers;
use Tests\TestCase;
use Mockery as m;
class RestfulControllerTest extends TestCase
{
    public function testTraits()
    {
        $controller = m::mock('Matthewbdaly\Harmony\Http\Controllers\RestfulController')->makePartial();
        $this->assertParentHasTrait('Illuminate\Foundation\Bus\DispatchesJobs', $controller);
        $this->assertParentHasTrait('Illuminate\Foundation\Validation\ValidatesRequests', $controller);
        $this->assertParentHasTrait('Illuminate\Foundation\Auth\Access\AuthorizesRequests', $controller);
    }
}

Actually implementing the assertion is fairly straightforward. You simply add the assertion as a method on the base test case you’re using. and accept whatever arguments are required, plus a final argument of $message = ''. Then you call self::assertThat(), as demonstrated below:

    public function assertParentHasTrait($trait, $class, $message = '')
    {
        $parent = get_parent_class($class);
        $traits = class_uses($parent);
        self::assertThat(in_array($trait, $traits), self::isTrue(), $message);
    }

In this case we’re asserting that the specified trait appears in the list of traits on the parent class. Note the use of self::isTrue() - this just verifies that the response is truthy.

Using this method it’s quite easy to create custom assertions, which can help make your tests less verbose and easier to read.

While PHPUnit is my normal go-to PHP testing framework, for some applications I find PHPSpec superior, in particular REST API clients. I’ve found that it makes for a better flow when doing test-driven development, because it makes it very natural to write a test first, then run it, then make the test place.

In this tutorial I’ll show you how to build a lookup API client for UK postcodes. In the process of doing so, we’ll use PHPSpec to drive our development process. We’ll also use HTTPlug as our underlying HTTP library. The advantage of this over using something like Guzzle is that we give library users the freedom to choose the HTTP library they feel is most appropriate to their situation.

Background

If you’re unfamiliar with it, the UK postcode system is our equivalent of a zip code in the US, but with two major differences:

• The codes themselves are alphanumeric instead of numeric, with the first part including one or two letters usually (but not always) derived from the nearest large town or city (eg L for Liverpool, B for Birmingham, OX for Oxford), or for London, based on the part of the city (eg NW for the north-west of London)
• A full postcode is in two parts (eg NW1 8TQ), and the first part narrows the location down to a similar area to a US-style zip code, while the second part usually narrows it down to a street (although sometimes large organisations that receive a lot of mail will have a postcode to themselves).

This means that if you have someone’s postcode and house name or address, you can use those details to look up the rest of their address details. This obviously makes it easier for users to fill out a form, such as when placing an order on an e-commerce site - you can just request those two details and then autofill the rest from them.

Unfortunately, it’s not quite that simple. The data is owned by Royal Mail, and they charge through the nose for access to the raw data, which places this data well outside the budgets of many web app developers. Fortunately, Ideal Postcodes offer a REST API for querying this data. It’s not free, but at 2.5p per request it’s not going to break the bank unless used excessively, and they offer some dummy postcodes that are free to query, which is perfectly fine for testing.

For those of you outside the UK, this may not be of much immediate use, but the underlying principles will still be useful, and you can probably build a similar client for your own nation’s postal code system. For instance, there’s a Zipcode API that those of you in the US can use, and if you understand what’s going on here it shouldn’t be hard to adapt it to work with that. If you do produce a similar client for your country’s postal code system, submit a pull request to update the README with a link to it and I’ll include it.

Setting up

First we’ll create a composer.json to specify our dependencies:

{
    "name": "matthewbdaly/postcode-client",
    "description": "A postcode lookup client.",
    "type": "library",
    "keywords": ["postcode"],
    "require": {
        "psr/http-message": "^1.0",
        "php-http/client-implementation": "^1.0",
        "php-http/httplug": "^1.0",
        "php-http/message-factory": "^1.0",
        "php-http/discovery": "^1.0"
    },
    "require-dev": {
        "psy/psysh": "^0.8.0",
        "phpspec/phpspec": "^3.2",
        "squizlabs/php_codesniffer": "^2.7",
        "php-http/mock-client": "^1.0",
        "php-http/message": "^1.0",
        "guzzlehttp/psr7": "^1.0"
    },
    "license": "MIT",
    "authors": [
        {
            "name": "Matthew Daly",
            "email": "matthewbdaly@gmail.com"
        }
    ],
    "autoload": {
        "psr-4": {
            "Matthewbdaly\\Postcode\\": "src/"
        }
    }
}

Note that we don’t install an actual HTTPlug client, other than the mock one, which is only useful for testing. This is deliberate - we’re giving developers working with this library the choice of working with whatever HTTP client they see fit. We do use the Guzzle PSR7 library, but that’s just for the PSR7 library.

Then we install our dependencies:

$ composer install

We also need to tell PHPSpec what our namespace will be. Save this as phpspec.yml:

suites:
    test_suite:
        namespace: Matthewbdaly\Postcode
        psr4_prefix: Matthewbdaly\Postcode

Don’t forget to update the namespace in both files to whatever you’re using, which should have a vendor name and a package name.

With that done, it’s time to introduce the next component.

Introducing HTTPlug

In the past I’ve usually used either Curl or Guzzle to carry out HTTP requests. However, the problem with this approach is that you’re forcing whoever uses your library to use whatever HTTP client, and whatever version of that client, that you deem appropriate. If they’re also using another library that someone else has written and they made different choices, you could have problems.

HTTPlug is an excellent way of solving this problem. By requiring only an interface and not a concrete implementation, using HTTPlug means that you can specify that the consumer of the library must provide a suitable implementation of that library, but leave the choice of implementation up to them. This means that they can choose whatever implementation best fits their use case. There are adapters for many different clients, so it’s unlikely that they won’t be able to find one that meets their needs.

In addition, HTTPlug provides the means to automatically determine what HTTP client to use, so that if one is not explicitly provided, it can be resolved without any action on the part of the developer. As long as a suitable HTTP adapter is installed, it will be used.

Getting started

One advantage of PHPSpec is that it will automatically generate much of the boilerplate for our client and specs. To create our client spec, run this command:

$ vendor/bin/phpspec desc Matthewbdaly/Postcode/Client
Specification for Matthewbdaly\Postcode\Client created in /home/matthew/Projects/postcode-client/spec/ClientSpec.php.

Now that we have a spec for our client, we can generate the client itself:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                    
  11  - it is initializable
      class Matthewbdaly\Postcode\Client does not exist.
                                      100%                                       1
1 specs
1 example (1 broken)
14ms
  Do you want me to create `Matthewbdaly\Postcode\Client` for you?              
                                                                         [Y/n] 
y
Class Matthewbdaly\Postcode\Client created in /home/matthew/Projects/postcode-client/src/Client.php.
                                      100%                                       1
1 specs
1 example (1 passed)
16ms

You will need to enter Y when prompted. We now have an empty class for our client.

Next, we need to make sure that the constructor for our client accepts two parameters:

• The HTTP client
• A message factory instance, which is used to create the request

Amend spec/ClientSpec.php as follows:

<?php
namespace spec\Matthewbdaly\Postcode;
use Matthewbdaly\Postcode\Client;
use PhpSpec\ObjectBehavior;
use Prophecy\Argument;
use Http\Client\HttpClient;
use Http\Message\MessageFactory;
class ClientSpec extends ObjectBehavior
{
    function let (HttpClient $client, MessageFactory $messageFactory)
    {
        $this->beConstructedWith($client, $messageFactory);
    }
    function it_is_initializable()
    {
        $this->shouldHaveType(Client::class);
    }
}

Note the use of the let() method here. This lets us specify how the object is constructed, with the beConstructedWith() method. Also, note that $this refers not to the test, but to the object being tested - this takes a bit of getting used to if you’re used to working with PHPUnit.

Also, note that the objects passed through are not actual instances of those objects - instead they are mocks created automatically by PHPSpec. This makes mocking extremely easy, and you can easily set up your own expectations on those mock objects in the test. If you want to use a real object, you can instantiate it in the spec as usual. If we need any other mocks, we can typehint them in our method in exactly the same way.

If we once again use vendor/bin/phpspec run we can now generate a constructor:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                    
  18  - it is initializable
      method Matthewbdaly\Postcode\Client::__construct not found.
                                      100%                                       1
1 specs
1 example (1 broken)
281ms
  Do you want me to create `Matthewbdaly\Postcode\Client::__construct()` for    
  you?                                                                          
                                                                         [Y/n] 
y
  Method Matthewbdaly\Postcode\Client::__construct() has been created.
                                      100%                                       1
1 specs
1 example (1 passed)
50ms

This will only create a placeholder for the constructor. You need to populate it yourself, so update src/Client.php as follows:

<?php
namespace Matthewbdaly\Postcode;
use Http\Client\HttpClient;
use Http\Discovery\HttpClientDiscovery;
use Http\Message\MessageFactory;
use Http\Discovery\MessageFactoryDiscovery;
class Client
{
    public function __construct(HttpClient $client = null, MessageFactory $messageFactory = null)
    {
        $this->client = $client ?: HttpClientDiscovery::find();
        $this->messageFactory = $messageFactory ?: MessageFactoryDiscovery::find();
    }
}

A little explanation is called for here. We need two arguments in our construct:

• An instance of Http\Client\HttpClient to send the request
• An instance of Http\Message\MessageFactory to create the request

However, we don’t want to force the user to create one. Therefore if they are not set, we use Http\Discovery\HttpClientDiscovery and Http\Discovery\MessageFactoryDiscovery to create them for us.

If we re-run PHPSpec, it should now pass:

$ vendor/bin/phpspec run
                                      100%                                       1
1 specs
1 example (1 passed)
31ms

Next, we want to have a method for retrieving the endpoint. Add the following method to spec/ClientSpec.php:

    function it_can_retrieve_the_base_url()
    {
        $this->getBaseUrl()->shouldReturn('https://api.ideal-postcodes.co.uk/v1/postcodes/');
    }

Here we’re asserting that fetching the base URL returns the given result. Note how much simpler and more intuitive this syntax is than PHPUnit would be:

$this->assertEquals('https://api.ideal-postcodes.co.uk/v1/postcodes/', $client->getBaseUrl());

Running the tests again should prompt us to create the boilerplate for the new method:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                      
  23  - it can retrieve the base url
      method Matthewbdaly\Postcode\Client::getBaseUrl not found.
                  50%                                     50%                    2
1 specs
2 examples (1 passed, 1 broken)
40ms
  Do you want me to create `Matthewbdaly\Postcode\Client::getBaseUrl()` for     
  you?                                                                          
                                                                         [Y/n] 
y
  Method Matthewbdaly\Postcode\Client::getBaseUrl() has been created.
Matthewbdaly/Postcode/Client                                                      
  23  - it can retrieve the base url
      expected "https://api.ideal-postcod...", but got null.
                  50%                                     50%                    2
1 specs
2 examples (1 passed, 1 failed)
72ms

Now we need to update that method to work as expected:

    protected $baseUrl = 'https://api.ideal-postcodes.co.uk/v1/postcodes/';
     ...
    public function getBaseUrl()
    {
        return $this->baseUrl;
    }

This should make the tests pass:

$ vendor/bin/phpspec run
                                      100%                                       2
1 specs
2 examples (2 passed)
34ms

Next, we need to be able to get and set the API key. Add the following to spec/ClientSpec.php:

    function it_can_get_and_set_the_key()
    {
        $this->getKey()->shouldReturn(null);
        $this->setKey('foo')->shouldReturn($this);
        $this->getKey()->shouldReturn('foo');
    }

Note that we expect $this->setKey('foo') to return $this. This is an example of a fluent interface - by returning an instance of the object, it enables methods to be chained, eg $client->setKey('foo')->get(). Obviously it won’t work for anything that has to return a value, but it’s a useful way of making your classes more intuitive to use.

Next, run the tests again and agree to the prompts as before:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                      
  28  - it can get and set the key
      method Matthewbdaly\Postcode\Client::getKey not found.
                         66%                                     33%             3
1 specs
3 examples (2 passed, 1 broken)
51ms
  Do you want me to create `Matthewbdaly\Postcode\Client::getKey()` for you?    
                                                                         [Y/n] 
y
  Method Matthewbdaly\Postcode\Client::getKey() has been created.
Matthewbdaly/Postcode/Client                                                      
  28  - it can get and set the key
      method Matthewbdaly\Postcode\Client::setKey not found.
                         66%                                     33%             3
1 specs
3 examples (2 passed, 1 broken)
43ms
  Do you want me to create `Matthewbdaly\Postcode\Client::setKey()` for you?    
                                                                         [Y/n] 
y
  Method Matthewbdaly\Postcode\Client::setKey() has been created.
Matthewbdaly/Postcode/Client                                                      
  28  - it can get and set the key
      expected [obj:Matthewbdaly\Postcode\Client], but got null.
                         66%                                     33%             3
1 specs
3 examples (2 passed, 1 failed)
52ms

Next, add our getter and setter for the key, as well as declaring the property $key:

    protected $key;
    public function getKey()
    {
        return $this->key;
    }
    public function setKey(string $key)
    {
        $this->key = $key;
        return $this;
    }

That should make the tests pass:

$ vendor/bin/phpspec run
                                      100%                                       3
1 specs
3 examples (3 passed)
38ms

With that done, our final task is to be able to handle sending requests. Add the following imports at the top of spec/ClientSpec.php:

use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\StreamInterface;

And add the following method at the bottom of the same file:

    function it_can_send_the_request(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)
    {
        $this->beConstructedWith($client, $messageFactory);
        $this->setKey('foo');
        $data = json_encode([
            'result' => [
                "postcode" => "SW1A 2AA",
                "postcode_inward" => "2AA",
                "postcode_outward" => "SW1A",
                "post_town" => "LONDON",
                "dependant_locality" => "",
                "double_dependant_locality" => "",
                "thoroughfare" => "Downing Street",
                "dependant_thoroughfare" => "",
                "building_number" => "10",
                "building_name" => "",
                "sub_building_name" => "",
                "po_box" => "",
                "department_name" => "",
                "organisation_name" => "Prime Minister & First Lord Of The Treasury",
                "udprn" => 23747771,
                "umprn" => "",
                "postcode_type" => "L",
                "su_organisation_indicator" => "",
                "delivery_point_suffix" => "1A",
                "line_1" => "Prime Minister & First Lord Of The Treasury",
                "line_2" => "10 Downing Street",
                "line_3" => "",
                "premise" => "10",
                "longitude" => -0.127695242183412,
                "latitude" => 51.5035398826274,
                "eastings" => 530047,
                "northings" => 179951,
                "country" => "England",
                "traditional_county" => "Greater London",
                "administrative_county" => "",
                "postal_county" => "London",
                "county" => "London",
            ]
        ]);
        $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);
        $client->sendRequest($request)->willReturn($response);
        $response->getStatusCode()->willReturn(200);
        $response->getBody()->willReturn($stream);
        $stream->getContents()->willReturn($data);
        $this->get('SW1A 2AA')->shouldBeLike(json_decode($data, true));
    }

This test is by far the biggest so far, so it merits some degree of explanation.

Note that we don’t make a real HTTP request against the API. This may sound strange, but bear with me. We have no control whatsoever over that API, and it could in theory become inaccessible or be subject to breaking changes at any time. We also don’t want to be shelling out for a paid service just to test our API client works. All we can do is test that our implementation will send the request we expect it to send - we don’t want our test suite reporting a bug when the API goes down.

We therefore typehint not just the dependencies for the constructor, but a request, response and stream instance. We mock our our responses from those instances using the willReturn() method, so we have complete control over what we pass to our client. That way we can return any appropriate response or throw any exception we deem fit to test the behaviour under those circumstances. For the message factory, we specify what arguments it should receive to create the request, and return our mocked-out request object.

Also, note we use shouldBeLike() to verify the response - this is effectively using the == operator, whereas shouldBe() uses the === operator, making it stricter.

Let’s run the tests, and don’t forget the prompt:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                      
  38  - it can send the request
      method Matthewbdaly\Postcode\Client::get not found.
                            75%                                     25%          4
1 specs
4 examples (3 passed, 1 broken)
55ms
  Do you want me to create `Matthewbdaly\Postcode\Client::get()` for you?       
                                                                         [Y/n] 
y
  Method Matthewbdaly\Postcode\Client::get() has been created.
Matthewbdaly/Postcode/Client                                                      
  38  - it can send the request
      expected [array:1], but got null.
                            75%                                     25%          4
1 specs
4 examples (3 passed, 1 failed)
56ms

Now we can implement the get() method:

    public function get(string $postcode)
    {
        $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([
            'api_key' => $this->getKey()
        ]);
        $request = $this->messageFactory->createRequest(
            'GET',
            $url,
            [],
            null,
            '1.1'
        );
        $response = $this->client->sendRequest($request);
        $data = json_decode($response->getBody()->getContents(), true);
        return $data;
    }

We first build up our URL, before using the message factory to create a request object. We then pass the built request to our client to send, before decoding the response into the format we want.

This should make our tests pass:

$ vendor/bin/phpspec run
                                      100%                                       4
1 specs
4 examples (4 passed)
307ms

Our client now works, but there are a couple of situations we need to account for. First, the API will raise a 402 if you make a request for a real postcode without having paid. We need to catch this and throw an exception. Add this to spec/ClientSpec.php:

use Matthewbdaly\Postcode\Exceptions\PaymentRequired;
    ...
    function it_throws_an_exception_if_payment_required(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)
    {
        $this->beConstructedWith($client, $messageFactory);
        $this->setKey('foo');
        $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);
        $client->sendRequest($request)->willReturn($response);
        $response->getStatusCode()->willReturn(402);
        $this->shouldThrow(PaymentRequired::class)->duringGet('SW1A 2AA');
    }

With that done, run the tests again:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                      
  87  - it throws an exception if payment required
      expected exception of class "Matthewbdaly\Postcode\Exc...", but got
      [exc:Prophecy\Exception\Call\UnexpectedCallException("Method call:
        - getBody()
      on Double\ResponseInterface\P15 was not expected, expected calls were:
        - getStatusCode()")].
                              80%                                     20%        5
1 specs
5 examples (4 passed, 1 failed)
130ms

Let’s amend the client to throw this exception:

use Matthewbdaly\Postcode\Exceptions\PaymentRequired;
    ...
    public function get(string $postcode)
    {
        $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([
            'api_key' => $this->getKey()
        ]);
        $request = $this->messageFactory->createRequest(
            'GET',
            $url,
            [],
            null,
            '1.1'
        );
        $response = $this->client->sendRequest($request);
        if ($response->getStatusCode() == 402) {
            throw new PaymentRequired;
        }
        $data = json_decode($response->getBody()->getContents(), true);
        return $data;
    }

And let’s re-run the tests:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                    
  87  - it throws an exception if payment required
      expected exception of class "Matthewbdaly\Postcode\Exc...", but got [obj:Error] with the
      message: "Class 'Matthewbdaly\Postcode\Exceptions\PaymentRequired' not found"
                              80%                                     20%        5
1 specs
5 examples (4 passed, 1 failed)
389ms

It fails now because the exception doesn’t exist. Let’s create it at src/Exceptions/PaymentRequired.php:

<?php
namespace Matthewbdaly\Postcode\Exceptions;
class PaymentRequired extends \Exception
{
}

That should be enough to make our tests pass:

$ vendor/bin/phpspec run
                                      100%                                       5
1 specs
5 examples (5 passed)
89ms

We also need to raise an exception when the postcode is not found, which raises a 404 error. Add the following spec:

use Matthewbdaly\Postcode\Exceptions\PostcodeNotFound;
    ...
    function it_throws_an_exception_if_postcode_not_found(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)
    {
        $this->beConstructedWith($client, $messageFactory);
        $this->setKey('foo');
        $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);
        $client->sendRequest($request)->willReturn($response);
        $response->getStatusCode()->willReturn(404);
        $this->shouldThrow(PostcodeNotFound::class)->duringGet('SW1A 2AA');
    }

Run the tests:

$ vendor/bin/phpspec run
Matthewbdaly/Postcode/Client                                                    
  98  - it throws an exception if postcode not found
      expected exception of class "Matthewbdaly\Postcode\Exc...", but got
      [exc:Prophecy\Exception\Call\UnexpectedCallException("Method call:
        - getBody()
      on Double\ResponseInterface\P20 was not expected, expected calls were:
        - getStatusCode()")].
                                83%                                     16%      6
1 specs
6 examples (5 passed, 1 failed)
538ms

This time we’ll create the exception class before updating the client. Create the following class at src/Exceptions/PostcodeNotFound.php:

<?php
namespace Matthewbdaly\Postcode\Exceptions;
/**
 * Postcode not found exception
 *
 */
class PostcodeNotFound extends \Exception
{
}

And update the client:

use Matthewbdaly\Postcode\Exceptions\PostcodeNotFound;
    ...
    public function get(string $postcode)
    {
        $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([
            'api_key' => $this->getKey()
        ]);
        $request = $this->messageFactory->createRequest(
            'GET',
            $url,
            [],
            null,
            '1.1'
        );
        $response = $this->client->sendRequest($request);
        if ($response->getStatusCode() == 402) {
            throw new PaymentRequired;
        }
        if ($response->getStatusCode() == 404) {
            throw new PostcodeNotFound;
        }
        $data = json_decode($response->getBody()->getContents(), true);
        return $data;
    }

Re-run the tests:

$ vendor/bin/phpspec run
                                      100%                                       6
1 specs
6 examples (6 passed)
103ms

And our API client is feature complete! You can find the source code of the finished client here.

Summary

Personally, I find that while PHPSpec isn’t appropriate for every use case, it’s particularly handy for API clients and it’s generally my go-to testing solution for them. It handles producing a lot of the boilerplate for me, and it results in a much better workflow for test-driven development as it makes it very natural to write the test first, then make it pass.

HTTPlug has been a revelation for me. While it takes a bit of getting used to if you’re used to something like Guzzle, it means that you’re giving consumers of your library the freedom to choose the HTTP client of their choice, meaning they don’t have to fight with several different libraries requiring different versions of Guzzle. It also allows for easy resolution of the HTTP client, rather than having to explicitly pass through an instance when instantiating your client. I’m planning to use it extensively in the future.

I’ve touched on using PostgreSQL to implement fuzzy search with Laravel before, but another type of search that PostgreSQL can handle fairly easily is full-text search. Here I’ll show you how to use it in a Laravel application.

An obvious use case for this kind of search is a personal blogging engine. It’s unlikely something like this is going to have enough content for it to be worth using a heavier solution like Elasticsearch, but a LIKE or ILIKE statement doesn’t really cut it either, so Postgres’s full text search is a good fit. Below you’ll see a Laravel migration for the blog posts table:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreatePostsTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('posts', function (Blueprint $table) {
            $table->increments('id');
            $table->string('title');
            $table->datetime('pub_date');
            $table->text('text');
            $table->string('slug');
            $table->integer('author_id');
            $table->timestamps();
        });
        DB::statement("ALTER TABLE posts ADD COLUMN searchtext TSVECTOR");
        DB::statement("UPDATE posts SET searchtext = to_tsvector('english', title || '' || text)");
        DB::statement("CREATE INDEX searchtext_gin ON posts USING GIN(searchtext)");
        DB::statement("CREATE TRIGGER ts_searchtext BEFORE INSERT OR UPDATE ON posts FOR EACH ROW EXECUTE PROCEDURE tsvector_update_trigger('searchtext', 'pg_catalog.english', 'title', 'text')");
    }
    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        DB::statement("DROP TRIGGER IF EXISTS tsvector_update_trigger ON posts");
        DB::statement("DROP INDEX IF EXISTS searchtext_gin");
        DB::statement("ALTER TABLE posts DROP COLUMN searchtext");
        Schema::dropIfExists('posts');
    }
}

Note that after we create the basic layout of our posts table, we then have to drop down to raw DB statements to achieve the next steps:

• We add a column called searchtext with a type of TSVECTOR (unfortunately Laravel doesn’t have a convenient method to create this column type, so we need to do it with a raw statement). This column will hold our searchable document.
• We use the to_tsvector() method to generate a document on each row that combines the title and text fields and store it in the searchtext column. Note also that we specify the language as the first argument. This is because Postgres’s full text search understands so-called “stopwords”, which are words that are so common as to not be worth bothering with at all, such as “the” - these will obviously differ between languages, so it’s prudent to explicitly state this so Postgres knows what stopwords to expect.
• We create a GIN index on the posts table using our new searchtext column.
• Finally we create a trigger which, when the table is amended, regenerates the search text.

With that done, we can now look at actually performing a full-text search. To facilitate easy re-use, we’ll create a local scope on our Post model. If you haven’t used scopes in Laravel before, they essentially allow you to break queries into reusable chunks. In this case, we expect our scope to receive two arguments, the query instance (which is passed through automatically), and the search text:

<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
class Post extends Model
{
    protected $fillable = [
        'title',
        'pub_date',
        'text',
        'slug',
        'author_id'
    ];
    public function scopeSearch($query, $search)
    {
        if (!$search) {
            return $query;
        }
        return $query->whereRaw('searchtext @@ to_tsquery(\'english\', ?)', [$search])
            ->orderByRaw('ts_rank(searchtext, to_tsquery(\'english\', ?)) DESC', [$search]);
    }
}

If $search is empty, we just return the query object as is. Otherwise, we first of all construct a WHERE clause that matches our search text against the searchtext column. Note the syntax used here:

searchtext @@ to_tsquery('english', 'foo')

We use the to_tsquery() method to match our text against our search document. As before, note that we specify the language.

Finally, we specify an order - we want the highest ranked matches to appear first, and this section of the query does that:

ts_rank(searchtext, to_tsquery('english', 'foo')) DESC

Here we use ts_rank() to ensure we get our results in the appropriate order. Note that for both queries, we passed the arguments through as parameterized queries, rather than constructing a raw string - we have to watch out for SQL injection when we’re writing raw queries, but we can use PDO’s parameterized queries from Eloquent in a raw statement, which makes things a bit easier.

Now we can call our new search scope as follows:

$posts = Post::search($search)->get();

Because the scope receives and returns a query builder instance, you can continue to add the rest of your query, or paginate it, as necessary:

$posts = Post::search($search)->where('draft', false)->simplePaginate(5);

If you’re working in a language that makes heavy use of accents, such as French, you might also want to install the unaccent extension (you can do this in the migration with CREATE EXTENSION unaccent). Then, any time we call to_tsvector(), you should pass any strings through the unaccent() method to strip out the accents.

Do we need the migrations?

Technically, we could do without the additional changes to the database structure - we could create a document on the fly inside a subquery and use that to query against, which would look something like this in SQL:

SELECT *
FROM
  (SELECT *,
          to_tsvector('english', posts.title) || to_tsvector('english', posts.text) AS document
   FROM "posts") search
WHERE search.document @@ to_tsquery('Redis')
ORDER BY ts_rank(search.document, to_tsquery('english', 'Redis')) DESC;

However, the performance is likely to be significantly worse using this approach as it has to recreate the document, and doesn’t have an existing index to query against. It’s also a pig to write something like this with an ORM.

I’m currently working on a more generic solution for implementing full text search with Postgres and Laravel, however so far it looks like that solution will not only be considerably more complex than this (consistently producing a suitable query for unknown data is rather fiddly), but you can’t create a column for the vector ahead of time, meaning the query will be slower. This approach, while it requires more work than simply installing a package, is not terribly hard to implement on a per-model basis and is easy to customise for your use case.

For many applications, using UUID’s as the primary keys on a database table can make a lot of sense. For mobile or offline apps, in particular, they mean you can create new objects locally and assign them a primary key without having to worry about it colliding with another object that was created in the meantime once it gets synchronised to the server. Also, they are less informative to nefarious users - an autoincrementing value in a URL tells a user that that value is the primary key, and means the app may potentially allow gathering of information via user enumeration (eg calling /api/v1/users/1, /api/v1/users/2 etc).

It’s fairly straightforward to use UUID’s as primary keys on your models when using PostgreSQL. First, you need to set up your migrations to use the uuid-ossp extension and set up the id field as both a UUID and the primary key. You also need to set a default value manually so that if it’s left empty it will generate a UUID for it.

DB::statement('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";');
Schema::create('items', function (Blueprint $table) {
    $table->uuid('id')->primary();
    $table->text('text')->nullable();
    $table->timestamps();
});
DB::statement('ALTER TABLE items ALTER COLUMN id SET DEFAULT uuid_generate_v4();');

Then, in the model definition, you need to tell Laravel to cast the id field to a string, and explicitly set the primary key to id:

class Item extends Model
{
    protected $casts = [
        'id' => 'string',
    ];
    protected $primaryKey = "id";
}

Once this is done, the model should generate the primary keys for you as usual, except as UUID’s. If your application needs to accept UUID primary keys that were created offline, such as in a mobile app, you will probably want to add the id field to the $fillable array on the model to allow this.

While the documentation for creating Artisan tasks is generally pretty good, it doesn’t really touch on creating tasks that generate new files. The only way to figure it out was to go digging through the source code. In this case, I was building an Artisan command to create Fractal transformers as part of a package I’m working on.

There’s a specialised class for generating files at Illuminate\Console\GeneratorCommand, which your command class should extend instead of Illuminate\Console\Command. In addition to the usual properties such as the signature and description, you also need to specify $type to give the type of class being generated. Also, note that the constructor is different, so if you use php artisan make:console to create the boilerplate for this command, you’ll need to delete the constructor.

<?php
namespace Matthewbdaly\MyPackage\Console\Commands;
use Illuminate\Console\GeneratorCommand;
use Symfony\Component\Console\Input\InputArgument;
class TransformerMakeCommand extends GeneratorCommand
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'make:transformer {name : The required name of the transformer class}';
    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Create a Fractal transformer';
    /**
     * The type of class being generated.
     *
     * @var string
     */
    protected $type = 'Fractal transformer';
    /**
     * Get the stub file for the generator.
     *
     * @return string
     */
    protected function getStub()
    {
        return __DIR__.'/stubs/transformer.stub';
    }
    /**
     * Get the console command arguments.
     *
     * @return array
     */
    protected function getArguments()
    {
        return [
            ['name', InputArgument::REQUIRED, 'The name of the command.'],
        ];
    }
    /**
     * Get the default namespace for the class.
     *
     * @param  string  $rootNamespace
     * @return string
     */
    protected function getDefaultNamespace($rootNamespace)
    {
        return $rootNamespace.'\Transformers';
    }
}

Note the getDefaultNamespace() method. If your class will live directly under the app folder this is not necessary. Otherwise, it needs to return the root namespace, with the folder structure you want after it. Here my class will live under app\Transformers, so I’ve set it to reflect that.

Also, note the getStub() method. This tells Artisan that it should use the specified stub file as the basis for our class. Below you’ll find the stub file I used for my transformer:

<?php
namespace DummyNamespace;
use Matthewbdaly\MyPackage\Transformers\BaseTransformer;
use Illuminate\Database\Eloquent\Model;
class DummyClass extends BaseTransformer
{
    public function transform(Model $model)
    {
        return [
            'id'            => (int) $model->id,
        ];
    }
}

Note that the DummyNamespace and DummyClass fields will be overwritten with the correct values.

Once this Artisan command is registered in the usual way, you can then run it as follows:

$ php artisan make:transformer Example

And it will generate a boilerplate class something like this:

<?php
namespace App\Transformers;
use Matthewbdaly\MyPackage\Transformers\BaseTransformer;
use Illuminate\Database\Eloquent\Model;
class Example extends BaseTransformer
{
    public function transform(Model $model)
    {
        return [
            'id'            => (int) $model->id,
        ];
    }
}

You can then replace the model with your own one as necessary, and add any further content to this class.

Recently I’ve been building and publishing a significant number of Laravel packages, and I thought I’d share details of some of them over the next few days.

Artisan Standalone is a package that, when installed in a standalone Laravel package (eg, not in an actual Laravel install, but in a package that you’re building that is intended for use with Laravel), allows you to use Artisan. It’s intended largely to make it quicker and easier to build functionality as separate packages by giving you access to the same generator commands as you have when working with a Laravel application. It came about largely from a need to scratch my own itch, as when building packages I was having to either run Artisan commands in a Laravel app and move them over, or copy them from existing files, which was obviously a pain in the proverbial.

You can install it with the following command:

$ composer require --dev matthewbdaly/artisan-standalone

Once it’s installed, you can access Artisan as follows:

$ vendor/bin/artisan

Note that it doesn’t explicitly include Laravel as a dependency - you’ll need to add that in the parent package to pull in the libraries it needs (which you should be doing anyway). It’s possible that there are some commands that won’t work in this context, but they’re almost certainly ones you won’t need here, such as the migrate command. As far as I can tell the generator commands, which are the only ones we’re really interested in here, all work OK.

The second package I’ve been working on recently is Laravel Package Boilerplate. It’s a basic starter boilerplate for building your own Laravel packages.

It’s not meant to be installed as a project dependency. Instead, run the following command to create a new project boilerplate with it:

composer create-project --prefer-dist matthewbdaly/laravel-package-boilerplate <YOUR_NEW_PACKAGE_DIRECTORY>

This will create a new folder that includes a src folder containing a service provider, and a tests folder containing a preconfigured base test case, as well as a simple test case for tests that don’t need the full application instantiated, in order to help keep your test suite as fast as possible.

In addition, it includes configuration files for:

• PHPUnit
• PHP CodeSniffer
• Travis CI

That way you can start your project off the right way with very little effort.

I’ve also added my Artisan Standalone project as a dependency - that way you can access any Artisan commands you need to generate files you need as follows:

$ vendor/bin/artisan

Hopefully this package should make it a lot easier to create new Laravel packages in future.

Most web apps have at least some need for some additional flat pages, for purposes such as:

• Terms and conditions
• Cookie/privacy policy
• FAQ

You can of course hard-code this content in a view file, but if this content is likely to change often it may be useful to give the site owners the capability to manage this themselves.

Laravel Flatpages is a package I wrote that adds a flatpage model, controller and view to your application. It’s loosely inspired by Django’s flatpages app. Using it, you can quickly and easily build a very simple brochure-style CMS. Each page contains fields for the title, content, slug, and an optional template field that specifies which view to use.

Note that it doesn’t include any kind of admin functionality, so you’ll need to add this yourself or find a package for it. It uses my repositories package to access the database, and this has caching built in, so when you create, update or delete a flatpage, you should either resolve Matthewbdaly\LaravelFlatpages\Contracts\Repositories\Flatpage and use the methods on that to make the changes (in which case the appropriate caches should be flushed automatically), or flush the cache. It also requires a cache backend that supports tags, such as Memcached or Redis.

It does not include routing in the package itself because I couldn’t find a way to guarantee that it would always be the last route, so instead you should put this in your routes/web.php and make sure it’s always the last route:

Route::get('{path}', '\Matthewbdaly\LaravelFlatpages\Http\Controllers\FlatpageController@page');

Otherwise you could wind up with problems. The reason for that is that it has to check the path against the slugs of the flat pages in the database, and if it doesn’t find any it raises a 404.

Or, if you prefer, you can use the middleware at Matthewbdaly\LaravelFlatpages\Http\Middleware\FlatpageMiddleware, which may be more convenient in many case. This should be added as the last global middleware in app\Http\Kernel.php.

When you first start doing test-driven development with Laravel, it can be quite hard to produce a test suite that runs quickly enough. The first time I used Laravel for a large project, I had a test suite that at one time, took over seven minutes to run, which was pretty awful considering that the ideal time for a test suite to take to run is no more than ten seconds.

Fortunately, with experience you can pick up some techniques which can quite drastically speed up your test suite. Here are some of the ones I’ve learned that can be useful.

Note that some of these are contradictory, and what works for one use case won’t necessarily work for another, so my advice is to try these and see what makes a difference for your use case.

Reduce the cost of hashing

Inside the createApplication() method of tests\CreatesApplication.php, place the following statement:

        Hash::setRounds(4);

This makes hashing passwords quicker and less demanding, and since you don’t care about the security of a password in a test, you’re not losing out in any way by doing so.

This, by itself, can massively reduce the time taken by your test suite - your mileage may vary, but I’ve personally seen it cut to a third of the previous time by using this. In fact, it’s recently been added to Laravel by default.

If you’re creating a lot of fixtures for tests, do so in a transaction

Sometimes, your application requires a lot of data to be added to the database just to be usable, and it’s quite common to use seeders for this purpose. However, it can take some time to insert a lot of data, especially if it has to be re-run for every test. If you do have to insert a lot of data before a test, you can cut down the time substantially by wrapping the seeder calls in a transaction:

<?php
use Illuminate\Database\Seeder;
use Illuminate\Database\Eloquent\Model;
use DB;
class DatabaseSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        Model::unguard();
        DB::beginTransaction();
        $this->call(GroupTableSeeder::class);
        $this->call(UserTableSeeder::class);
        $this->call(ProjectTableSeeder::class);
        DB::commit();
        Model::reguard();
    }
}
`

I’ve personally seen this trick cut the insert time by over half, every single time the database is seeded. If you don’t have much data to insert, it may not help, but for large amounts of data it can make a big difference.

If a lot of tests need the same data, migrate and seed it first, then wrap the tests in transactions and roll them back afterwards

If multiple tests need to work with the same dataset, you should consider running the migrations and seeders before the first test, and then wrapping each test inside a transaction. That way the data will only be inserted once, and will be rolled back to that initial good state after each test.

    protected static $migrated = false;
    public function setUp()
    {
        parent::setUp();
        DB::beginTransaction();
    }
    public function tearDown()
    {
        DB::rollback();
        parent::tearDown();
    }
    public static function setUpBeforeClass()
    {
        if (!self::$migrated) {
            Artisan::call('migrate:fresh');
            Artisan::call('db:seed');
            self::$migrated = true;
        }
    }

Using something like this instead of one of the existing testing traits may be a better fit under those circumstances. However, if your application uses transactions for some functionality this might cause problems.

Don’t create a full instance of the Laravel application unless you have to

Not every test requires that you instantiate the full Laravel application, and doing so slows your tests down. If you don’t absolutely need the full application instantiated in the test, consider having your test inherit from the below simple test case class instead:

<?php
namespace Tests;
use Mockery\Adapter\Phpunit\MockeryPHPUnitIntegration;
use PHPUnit\Framework\TestCase as BaseTestCase;
class SimpleTestCase extends BaseTestCase
{
    use MockeryPHPUnitIntegration;
}

For properly isolated unit tests, using this base class instead can have a noticeable effect on performance.

If you can, use an in-memory SQLite database for testing

This isn’t an option if you’re relying on the features of another database, but if it is, this is usually the fastest way to go. Configure it as follows in phpunit.xml:

        <env name="DB_CONNECTION" value="sqlite"/>
        <env name="DB_DATABASE" value=":memory:"/>

Use the new Refresh Database trait

   use RefreshDatabase;

This testing trait is generally more efficient than migrating down and up, because it empties the database afterwards rather than stepping through the changes of each migration. If you have a non-trivial number of migrations, it will almost certainly be quicker than migrating down, then back up for the next test.

Mock what you can’t control

You should never, ever be making calls to external APIs in your test suite, because you can’t control whether those external API’s work - if a third-party API goes down, you may get a failed test run even if your application is working perfectly, not to mention it will add the time taken to send the request and receive a response to the test time. Instead, mock the calls to the third-party API.

For large applications, consider moving parts into a separate package

If you have a particularly large application, it’s worth considering moving parts of it out into standalone packages and requiring them using Composer’s support for private Git repositories. That way, those packages can have their own test suites, and the main application’s test suite can cover the remaining functionality.

For instance, it’s fairly straightforward to pull out your models and migrations and put them in a separate package, and the tests for them can go with them to that package.

You should also consider whether parts of your application would be useful as standalone packages, and if so pull them out along with their tests. That way, not only are you making your test suite quicker, but you’re also saving yourself work by creating a reusable solution for a problem you might encounter again in the future.

Turn off XDebug

XDebug has a horrendous effect on the performance of the test suite. Turn it off unless you need it to generate test coverage. Better yet, set up continuous integration and have that generate the coverage for you.

Summary

When you first start using Laravel, it can be hard to keep your test suite lean, and the longer a test suite takes to run, the less likely it is to actually get run regularly. To practice TDD properly, your test suite should not take long enough that your mind starts to wander, and ten seconds is a good target to aim for in this regard - you need to be able to run it several times a minute without problem. Obviously things like having a faster computer or an SSD will help, but there’s a lot you can do to make your test suite more efficient, even when running on a quite basic machine.

Laravel Comments is a package I recently released that allows you to add comments to any model in your application. Possible models you could use it to enable comments on might include:

• Blog posts
• Forum posts
• Issues on an issue tracker

It’s loosely inspired by Django’s comments system.

Installation

Run this command to install it:

$ composer require matthewbdaly/laravel-comments

You will also need to run php artisan migrate to create the appropriate tables.

Making a model commentable

Add the following trait to a model to make it commentable:

Matthewbdaly\LaravelComments\Eloquent\Traits\Commentable

The comments table uses a polymorphic relation, so it should be possible to attach it to pretty much any model. The model should now have a comments relation, allowing you to get the comments for a model instance.

Displaying the comments

Obviously you can just render the comments in a view you can create yourself, but it’s usually going to be more convenient to use the existing view, even if just as a starting point, which includes the ability to submit new comments and flag existing ones. Include it in your views as follows:

@include('comments::comments', ['parent' => $post])

The argument passed to parent should be the model instance for which you want to display the comments form. Obviously, you can easily override this to use your own custom form instead.

The package also contains the following views:

• comments::commentsubmitted
• comments::flagsubmitted

These are basically just acknowledgement screens for when a comment has been submitted or flagged, and you’ll probably want to override them.

The package also has its own routes and controller included for submitting comments and flags.

Using the models directly

Of course there’s nothing stopping you creating your own routes and controllers for creating, viewing and flagging comments, and if, for instance, you wish to build a REST API that allows for adding comments to objects you can just use these models directly:

• Matthewbdaly\LaravelComments\Eloquent\Models\Comment
• Matthewbdaly\LaravelComments\Eloquent\Models\Comment\Flag

I recommend that you use my repositories, which are as follows:

• Matthewbdaly\LaravelComments\Contracts\Repositories\Comment
• Matthewbdaly\LaravelComments\Contracts\Repositories\Comment\Flag

These use matthewbdaly/laravel-repositories and so implement caching on the decorated repository, making it simple to ensure your models get cached appropriately. However, they aren’t compulsory.

Events

You can set up listeners for the following events:

• Matthewbdaly\LaravelComments\Events\CommentReceived

Fired when a new comment is submitted. The package does not include any kind of validation of comments, so you can instead listen for this event and implement your own functionality to validate them (eg, check it with Akismet, check for links). That way you can easily customise how it handles potentially spammy comments for your own particular use case.

• Matthewbdaly\LaravelComments\Events\CommentFlagged

This event indicates that a comment has been flagged for moderator attention. You can use this event to send whatever notification is most appropriate (eg, email, Slack, SMS).

When working with any Laravel application that implements authentication, you’ll need to set up a user account to be able to work with it. One way of doing that is to add a user in a seeder, but that’s only really suitable if every user is going to use the same details.

Instead, you may want to create an Artisan command to set up the user account. Here’s an example of a command that does that:

<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Hash;
class CreateUser extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'create:user';
    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Creates a single user';
    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }
    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        // Get user model from config
        $model = config('auth.providers.users.model');
        // Let user know what this will do
        $this->info('I\'ll ask you for the details I need to set up the user');
        // Get username
        $name = $this->ask('Please provide the username');
        // Get email
        $email = $this->ask('Please provide the email address');
        // Get password
        $password = $this->secret('Please provide the password');
        // Create model
        $user = new $model;
        $user->name = $name;
        $user->email = $email;
        $user->password = Hash::make($password);
        $user->save();
        $this->info('User saved');
    }
}

We fetch the user model from the config, before asking the user for the data we need. Then we insert it into the database and confirm it to the user.

Then we just need to register the command in App\Console\Kernel.php:

    protected $commands = [
        \App\Console\Commands\CreateUser::class,
    ];

And we can run our command with php artisan create:user.

Today I’ve been working on a generic, reusable Laravel admin interface, loosely inspired by the Django admin, that dynamically picks up the field types and generates an appropriate input field accordingly.

One problem I’ve run into is that getting a representation of a database table’s fields relies on doctrine/dbal, and its support for the more unusual PostgreSQL field types is spotty at best. I’ve been testing it out on a Laravel-based blogging engine, which has full-text search using the TSVECTOR field type, which isn’t supported, and it threw a nasty Unknown database type tsvector requested error.

Fortunately, it’s possible to register custom field type mappings easily enough. In this case we can safely treat a TSVECTOR field as a string` type anyway, so we can map it to the string type. We can do so in the boot method of a service provider:

<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
    /**
     * Bootstrap any application services.
     *
     * @return void
     */
    public function boot()
    {
        // Register the TSVECTOR column
        $conn = $this->app->make('Illuminate\Database\ConnectionInterface');
        $conn->getDoctrineSchemaManager()
            ->getDatabasePlatform()
            ->registerDoctrineTypeMapping('tsvector', 'string');
    }
    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
       //
    }
}

We register a Doctrine type mapping that maps the tsvector type to a string. Now Doctrine will just treat it as a string.

We can then retrieve the field types as follows:

        $table = $this->model->getTable();
        $fields = array_values(Schema::getColumnListing($table));
        $fielddata = [];
        foreach ($fields as $field){
            if ($field != 'id' && $field != 'created_at' && $field != 'updated_at' && $field != 'deleted_at') {
                try {
                    $fielddata[$field] = Schema::getColumnType($table, $field);
                } catch (\Exception $e) {
                    $fielddata[$field] = 'unknown';
                }
            }
        }

Note that we specifically don’t want to retrieve the ID or timestamps, so we exclude them - the user should never really have the need to update them manually. We fetch the table from the model and then call Schema::getColumnListing() to retrieve a list of fields for that table. Finally we call Schema::getColumnType() to actually get the type of each column.

Now, I suspect the performance of this admin interface is going to be inferior to a more specific one because it has to retrieve the fields all the time, but that’s not the point here - with a non-user facing admin interface, performance isn’t quite as much of an issue. For the same reason the admin doesn’t do any caching at all. It’s still useful under certain circumstances to be able to reverse-engineer the table structure and render an appropriate form dynamically.

Although helpers are an important part of Laravel, the documentation doesn’t really touch on creating them. Fortunately, doing so it fairly easy.

Here I’m building a helper for formatting dates for the HTML5 datetime-local form input. First we define the helper function in app\Helpers.php:

<?php
use Carbon\Carbon;
if (!function_exists('format_date')) {
    function format_date(string $date)
    {
        return Carbon::parse($date, config('app.timezone'))->format('Y-m-d\TH:i:s');
    }
}

Then we create a service provider to load them:

<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
class HelperServiceProvider extends ServiceProvider
{
    /**
     * Bootstrap the application services.
     *
     * @return void
     */
    public function boot()
    {
        //
    }
    /**
     * Register the application services.
     *
     * @return void
     */
    public function register()
    {
        //
        require_once app_path() . '/Helpers.php';
    }
}

Finally,we register the service provider in config/app.php:

    'providers' => [
       ...
        App\Providers\HelperServiceProvider::class,
   ],

Of course, once you have this all set up for one helper, it’s easy to add more because they can all go in app/Helpers.php.

Creating your own helpers is a good way of refactoring unwanted logic out of your Blade templates or controllers and making it more reusable and maintainable, particularly for things like formatting dates or strings.

Creating repositories for your Laravel models, as well as creating caching decorators for them, is a useful way of not only implementing caching in your web app, but decoupling the application from a specific ORM. Unfortunately, it can involve writing a fair amount of boilerplate code.

Laravel Repositories is a set of base classes and interfaces for creating repositories and decorators in your application. It consists of:

• A generic interface for repositories
• A base repository that implements the interface and can be extended for your own repositories
• A base decorator that also implements the interface and can similarly be extended

By using these, not only are you able to implement caching quickly and easily for most use cases, but you can easily extend the base classes to add additional methods for your use case. By creating new interfaces that extend the base interface, then having your repositories extend the repository and decorator, you can minimise the amount of work required to set up new repositories.

The main interface used is Matthewbdaly\LaravelRepositories\Repositories\Interfaces\AbstractRepositoryInterface, and your interfaces should extend this. Your decorators should extend Matthewbdaly\LaravelRepositories\Repositories\Decorators\BaseDecorator, and your repositories should extend Matthewbdaly\LaravelRepositories\Repositories\Base. Then, if you add any additional methods to your interface and ensure your repository and decorator implement that interface, it should be straightforward to type-hint the interface and get back the decorated repository, which will handle caching for you.

To be able to type-hint the repositories, you need to set them up in a service provider:

<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
    /**
     * Bootstrap any application services.
     *
     * @return void
     */
    public function boot()
    {
        //
    }
    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->singleton('App\Repositories\Interfaces\ExampleRepositoryInterface', function () {
            $baseRepo = new \App\Repositories\EloquentExampleRepository(new \App\Example);
            $cachingRepo = new \App\Repositories\Decorators\ExampleDecorator($baseRepo, $this->app['cache.store']);
            return $cachingRepo;
        });
    }
}

Also, note that the cache backend used must be one that supports tags, such as Redis or Memcached. Data is cached using a tag derived from the model name. This also means you have to be careful when eager-loading relations, as the data will be cached under the main model’s name, not that of the relation. You may want to set up separate model events to flush those tags when the related field is updated.

For the uninitiated, OpenSearch is the technology that lets you enter a site’s URL, and then press Tab to start searching on that site - you can see it in action on this site. It’s really useful, and quite easy to implement if you know how.

OpenSearch relies on having a particular XML file available. Here’s the opensearch.xml file for this site:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns:moz="http://www.mozilla.org/2006/browser/search/"
   xmlns="http://a9.com/-/spec/opensearch/1.1/">
   <ShortName>matthewdaly.co.uk</ShortName>
   <Description>Search matthewdaly.co.uk</Description>
   <InputEncoding>UTF-8</InputEncoding>
   <Url method="get" type="text/html"
      template="http://www.google.com/search?q={searchTerms}&amp;sitesearch=matthewdaly.co.uk"/>
</OpenSearchDescription>

In this case, as this site uses a static site generator I can’t really do the search on the site, so it’s handed off to a Google site-specific search, but the principle is the same. The three relevant fields are as follows:

• ShortName - The short name of the site (this should usually just be the domain name)
• Description - A human-readable description such as Search mysite.com
• Url - Specifies the HTTP method that should be used to search (GET or POST), and a template for the URL. The search is automatically inserted where {searchTerms} appears

A more typical example of the Url field might be as follows:

   <Url method="get" type="text/html"
      template="http://www.example.com/search?q={searchTerms}"/>

Normally you will be pointing the template to your site’s own search page. Note that OpenSearch doesn’t actually do any searching itself - it just tells your browser where to send your search request.

With that file saved as opensearch.xml, all you have to do is add it to the <head> in your HTML:

<link href="/opensearch.xml" rel="search" title="Search title" type="application/opensearchdescription+xml">

And that should be all you need to do to get OpenSearch working.

For Laravel sites, I’ve recently created a package for implementing Opensearch that should help as well. With that you need only install the package, and set the fields in the config to point at your existing search page, in order to get OpenSearch working.

If you have a Laravel application that requires users to log in and you use Clockwork or Laravel DebugBar to examine the queries that take place, you’ll probably notice a query that fetches the user model occurs quite a lot. This is because the user’s ID gets stored in the session, and is then used to retrieve the model.

This query is a good candidate for caching because not only is that query being made often, but it’s also not something that changes all that often. If you’re careful, it’s quite easy to set your application up to cache the user without having to worry about invalidating the cache.

Laravel allows you to define your own user providers in order to fetch the user’s details. These must implement Illuminate\Contracts\Auth\UserProvider and must return a user model from the identifier provided. Out of the box it comes with two implementations, Illuminate\Auth\EloquentUserProvider and Illuminate\Auth\DatabaseUserProvider, with the former being the default. Our caching user provider can extend the Eloquent one as follows:

<?php
namespace App\Auth;
use Illuminate\Auth\EloquentUserProvider;
use Illuminate\Contracts\Cache\Repository;
use Illuminate\Contracts\Hashing\Hasher as HasherContract;
class CachingUserProvider extends EloquentUserProvider
{
    /**
     * The cache instance.
     *
     * @var Repository
     */
    protected $cache;
    /**
     * Create a new database user provider.
     *
     * @param  \Illuminate\Contracts\Hashing\Hasher  $hasher
     * @param  string  $model
     * @param  Repository  $cache
     * @return void
     */
    public function __construct(HasherContract $hasher, $model, Repository $cache)
    {
        $this->model = $model;
        $this->hasher = $hasher;
        $this->cache = $cache;
    }
    /**
     * Retrieve a user by their unique identifier.
     *
     * @param  mixed  $identifier
     * @return \Illuminate\Contracts\Auth\Authenticatable|null
     */
    public function retrieveById($identifier)
    {
        return $this->cache->tags($this->getModel())->remember('user_by_id_'.$identifier, 60, function () use ($identifier) {
            return parent::retrieveById($identifier);
        });
    }
}

Note that we override the constructor to accept a cache instance as well as the other arguments. We also override the retrieveById() method to wrap a call to the parent’s implementation inside a callback that caches the response. I usually tag anything I cache with the model name, but if you need to use a cache backend that doesn’t support tagging this may not be an option. Our cache key also includes the identifier so that it’s unique to that user.

We then need to add our user provider to the auth service provider:

<?php
namespace App\Providers;
use Illuminate\Support\Facades\Gate;
use Illuminate\Foundation\Support\Providers\AuthServiceProvider as ServiceProvider;
use App\Auth\CachingUserProvider;
use Illuminate\Support\Facades\Auth;
class AuthServiceProvider extends ServiceProvider
{
    /**
     * Register any authentication / authorization services.
     *
     * @return void
     */
    public function boot()
    {
        $this->registerPolicies();
        Auth::provider('caching', function ($app, array $config) {
            return new CachingUserProvider(
                $app->make('Illuminate\Contracts\Hashing\Hasher'),
                $config['model'],
                $app->make('Illuminate\Contracts\Cache\Repository')
            );
        });
    }
}

Note here that we call this provider caching, and we pass it the hasher, the model name, and an instance of the cache. Then, we need to update config/auth.php to use this provider:

    'providers' => [
        'users' => [
            'driver' => 'caching',
            'model' => App\Eloquent\Models\User::class,
        ],
    ],

The only issue now is that our user models will continue to be cached, even when they are updated. To be able to flush the cache, we can create a model event that fires whenever the user model is updated:

<?php
namespace App\Eloquent\Models;
use Illuminate\Notifications\Notifiable;
use Illuminate\Foundation\Auth\User as Authenticatable;
use App\Events\UserAmended;
class User extends Authenticatable
{
    use Notifiable;
    protected $dispatchesEvents = [
        'saved' => UserAmended::class,
        'deleted' => UserAmended::class,
        'restored' => UserAmended::class,
    ];
}

This will call the UserAmended event when a user model is created, updated, deleted or restored. Then we can define that event:

<?php
namespace App\Events;
use Illuminate\Broadcasting\Channel;
use Illuminate\Queue\SerializesModels;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Broadcasting\PresenceChannel;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
use App\Eloquent\Models\User;
class UserAmended
{
    use Dispatchable, InteractsWithSockets, SerializesModels;
    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct(User $model)
    {
        $this->model = $model;
    }
}

Note our event contains an instance of the user model. Then we set up a listener to do the work of clearing the cache:

<?php
namespace App\Listeners;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use App\Events\UserAmended;
use Illuminate\Contracts\Cache\Repository;
class ClearUserId
{
    /**
     * Create the event listener.
     *
     * @return void
     */
    public function __construct(Repository $cache)
    {
        $this->cache = $cache;
    }
    /**
     * Handle the event.
     *
     * @param  object  $event
     * @return void
     */
    public function handle(UserAmended $event)
    {
        $this->cache->tags(get_class($event->model))->forget('user_by_id_'.$event->model->id);
    }
}

Here, we get the user model’s class again, and clear the cache entry for that user model.

Finally, we hook up the event and listener in the event service provider:

<?php
namespace App\Providers;
use Illuminate\Support\Facades\Event;
use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
class EventServiceProvider extends ServiceProvider
{
    /**
     * The event listener mappings for the application.
     *
     * @var array
     */
    protected $listen = [
        'App\Events\UserAmended' => [
            'App\Listeners\ClearUserId',
        ],
    ];
    /**
     * Register any events for your application.
     *
     * @return void
     */
    public function boot()
    {
        parent::boot();
        //
    }
}

With that done, our user should be cached after the first load, and flushed when the model is amended.

Handling eager-loaded data

It may be that you’re pulling in additional data from the user model in your application, such as roles, permissions, or a separate profile model. Under those circumstances it makes sense to treat that data in the same way by eager-loading it along with your user model.

<?php
namespace App\Auth;
use Illuminate\Auth\EloquentUserProvider;
use Illuminate\Contracts\Cache\Repository;
use Illuminate\Contracts\Hashing\Hasher as HasherContract;
class CachingUserProvider extends EloquentUserProvider
{
    /**
     * The cache instance.
     *
     * @var Repository
     */
    protected $cache;
    /**
     * Create a new database user provider.
     *
     * @param  \Illuminate\Contracts\Hashing\Hasher  $hasher
     * @param  string  $model
     * @param  Repository  $cache
     * @return void
     */
    public function __construct(HasherContract $hasher, $model, Repository $cache)
    {
        $this->model = $model;
        $this->hasher = $hasher;
        $this->cache = $cache;
    }
    /**
     * Retrieve a user by their unique identifier.
     *
     * @param  mixed  $identifier
     * @return \Illuminate\Contracts\Auth\Authenticatable|null
     */
    public function retrieveById($identifier)
    {
        return $this->cache->tags($this->getModel())->remember('user_by_id_'.$identifier, 60, function () use ($identifier) {
          $model = $this->createModel();
          return $model->newQuery()
            ->with('roles', 'permissions', 'profile')
            ->where($model->getAuthIdentifierName(), $identifier)
            ->first();
        });
    }
}

Because we need to amend the query itself, we can’t just defer to the parent implementation like we did above and must instead copy it over and amend it to eager-load the data.

You’ll also need to set up model events to clear the cache whenever one of the related fields is updated, but it should be fairly straightforward to do so.

Summary

Fetching a user model (and possibly some relations) on every page load while logged in can be a bit much, and it makes sense to cache as much as you can without risking serving stale data. Using this technique you can potentially cache a lot of repetitive, unnecessary queries and make your application faster.

This technique will also work in cases where you’re using other methods of maintaining user state, such as JWT, as long as you’re making use of a guard for authentication purposes, since all of these guards will still be using the same user provider. In fact, I first used this technique on a REST API that used JWT for authentication, and it’s worked well in that case.

Deployment processes have a nasty tendency to be a mish-mash of cobbled-together scripts or utilities in many web shops, with little or no consistency in practice between them. As a result, it’s all too easy for even the most experienced developer to mess up a deployment.

I personally have used all kinds of bodged-together solutions. For a while I used Envoy scripts to deploy my Laravel apps, but then there was an issue with the SSH library in PHP 7 that made it impractical to use it. Then I adopted Fabric, which I’d used before for deploying Django apps and will do fine for deploying PHP apps too, but it wasn’t much more sophisticated than using shell scripts for deployment purposes. There are third-party services like Deploybot, but these are normally quite expensive for what they are.

A while back I heard of Deployer, but I didn’t have the opportunity to try it until recently on a personal project as I was working somewhere that had its own in-house deployment process. It’s a PHP-specific deployment tool with recipes for deploying applications built with various frameworks and CMS’s, including Laravel, Symfony, CodeIgniter and Drupal.

Installing Deployer

Deployer is installed as a .phar file, much like you would with Composer:

$ curl -LO https://deployer.org/deployer.phar
$ mv deployer.phar /usr/local/bin/dep
$ chmod +x /usr/local/bin/dep

With that done, you should be able to run the following command in your project’s directory to create a Deployer script:

$ dep init

In response, you should see a list of project types:

  Welcome to the Deployer config generator  
 This utility will walk you through creating a deploy.php file.
 It only covers the most common items, and tries to guess sensible defaults.
 Press ^C at any time to quit.
 Please select your project type [Common]:
  [0] Common
  [1] Laravel
  [2] Symfony
  [3] Yii
  [4] Yii2 Basic App
  [5] Yii2 Advanced App
  [6] Zend Framework
  [7] CakePHP
  [8] CodeIgniter
  [9] Drupal
 >

Here I chose Laravel as I was deploying a Laravel project. I was then prompted for the repository URL - this will be filled in with the origin remote if the current folder is already a Git repository:

Repository [git@gitlab.com:Group/Project.git]:
 >

You’ll also see a message about contributing anonymous usage data. After answering this, the file deploy.php will be generated:

<?php
namespace Deployer;
require 'recipe/laravel.php';
// Configuration
set('repository', 'git@gitlab.com:Group/Project.git');
set('git_tty', true); // [Optional] Allocate tty for git on first deployment
add('shared_files', []);
add('shared_dirs', []);
add('writable_dirs', []);
// Hosts
host('project.com')
    ->stage('production')
    ->set('deploy_path', '/var/www/project.com');
host('beta.project.com')
    ->stage('beta')
    ->set('deploy_path', '/var/www/project.com');  
// Tasks
desc('Restart PHP-FPM service');
task('php-fpm:restart', function () {
    // The user must have rights for restart service
    // /etc/sudoers: username ALL=NOPASSWD:/bin/systemctl restart php-fpm.service
    run('sudo systemctl restart php-fpm.service');
});
after('deploy:symlink', 'php-fpm:restart');
// [Optional] if deploy fails automatically unlock.
after('deploy:failed', 'deploy:unlock');
// Migrate database before symlink new release.
before('deploy:symlink', 'artisan:migrate');

By default it has two hosts, beta and production, and you can refer to them by these names. You can also add or remove hosts, and amend the existing ones. Note the deploy path as well - this sets the place where the application gets deployed to.

Note that it’s set up to expect the server to be using PHP-FPM and Nginx by default, so if you’re using Apache you may need to amend the command to restart the server. Also, note that if like me you’re using PHP 7 on a distro like Debian that also has PHP 5 around, you’ll probably need to change the references to php-fpm as follows:

desc('Restart PHP-FPM service');
task('php-fpm:restart', function () {
    // The user must have rights for restart service
    // /etc/sudoers: username ALL=NOPASSWD:/bin/systemctl restart php-fpm.service
    run('sudo systemctl restart php7.0-fpm.service');
});
after('deploy:symlink', 'php-fpm:restart');

You will also need to make sure the acl package is installed - on Debian and Ubuntu you can install it as follows:

$ sudo apt-get install acl

Now, the recipe for deploying a Laravel app will include the following:

• Pulling from the Git remote
• Updating any Composer dependencies to match composer.json
• Running the migrations
• Optimizing the application

In addition, one really great feature Deployer offers is rollbacks. Rather than checking out your application directly into the project root you specify, it numbers each release and deploys it in a separate folder, before symlinking that folder to the project root as current. That way, if a release cannot be deployed successfully, rather than leaving your application in an unfinished state, Deployer will symlink the previous version so that you still have a working version of your application.

If you have configured Deployer for that project, you can deploy using the following command where production is the name of the host you’re deploying to:

$ dep deploy production

The output will look something like this:

✔ Executing task deploy:prepare
✔ Executing task deploy:lock
✔ Executing task deploy:release
➤ Executing task deploy:update_code
Counting objects: 761, done.
Compressing objects: 100% (313/313), done.
Writing objects: 100% (761/761), done.
Total 761 (delta 384), reused 757 (delta 380)
Connection to linklater.shellshocked.info closed.
✔ Ok
✔ Executing task deploy:shared
✔ Executing task deploy:vendors
✔ Executing task deploy:writable
✔ Executing task artisan:storage:link
✔ Executing task artisan:view:clear
✔ Executing task artisan:cache:clear
✔ Executing task artisan:config:cache
✔ Executing task artisan:optimize
✔ Executing task artisan:migrate
✔ Executing task deploy:symlink
✔ Executing task php-fpm:restart
✔ Executing task deploy:unlock
✔ Executing task cleanup
✔ Executing task success
Successfully deployed!

As you can see, we first of all lock the application and pull the latest version from the Git remote. Next we copy the files shared between releases (eg the .env file, the storage/ directory etc), update the dependencies, and make sure the permissions are correct. Next we link the storage, clear all the cached content, optimize our app, and migrate the database, before we set up the symlink. Finally we restart the web server and unlock the application.

In the event you discover a problem after deploy and need to rollback manually, you can do so with the following command:

$ dep rollback production

That makes it easy to ensure that in the event of something going wrong, you can quickly switch back to an earlier version with zero downtime.

Deployer has made deployments a lot less painful for me than any other solution I’ve tried. The support for rollbacks means that if something goes wrong it’s trivial to switch back to an earlier revision.

Skim through any programming-related forum and you’ll often find statements along the lines of the following:

• “I chose Lumen for my website because the benchmarks show it’s faster than Laravel”
• “I’m using raw queries because they’re faster than using an ORM”
• “I wrote the site in pure PHP to avoid the overhead of a framework”

Making my web apps performant is something I care deeply about. Yet every time I see something like this I cringe. Why? Because statements like these are full of wild misconceptions about the real performance bottlenecks in modern web applications. I don’t blame framework vendors for publishing benchmarks of their applications, since the performance of web apps is a big issue, but they are often misleading even when they’re correct, and it’s all too easy for inexperienced developers to think that performance is a matter of picking the fastest framework, rather than following a methodology of identifying and dealing with performance bottlenecks.

In this post I’ll explain why the performance of the framework, while not a non-issue, should come way down the list of factors involved in choosing a framework (or not to use one at all), behind functionality and developer productivity, and how many other factors not related to the choice of framework are involved.

Benchmarks don’t include real-world optimisations

When benchmarking a number of frameworks together, you’ll typically be testing some fairly basic behaviour such as rendering a view, and maybe making a database query. It’s rare for them to also include things such as caching queries or sending the correct HTTP caching headers.

Also, it’s quite common for the party creating the benchmark to have their own preference they’re more familiar with, in which case they’ll have a better idea of how to optimise that one. If they don’t know how to optimise all of them to the same extent, the end results is going to be biased. For example, in the case of Laravel, running php artisan optimize can significantly improve application performance by caching large chunks of the application.

In addition, the configuration for the web server is quite likely to be suboptimal compared to a production server. For instance, they may not have the opcode cache installed, or Nginx may not set the right headers on static assets. Under these circumstances the benchmarks are very likely to be misleading. Ultimately, if you chose to completely rewrite an entire application from scratch in a new framework to claw back a few milliseconds, how do you know you’ll actually see that translate into better performance in production for your particular use case?

And if you’re even considering running a supposedly performance-critical application on shared hosting, you should hang your head in shame…

Your from-scratch implementation of functionality is probably slower than an existing one

If you’re building some functionality from scratch instead of using an off-the-shelf library on the basis of performance, just stop. Existing libraries have usually had a great deal of attention already, should have working test suites, and depending on how active the developer community around them is, they may well have found and resolved the most egregious performance bottlenecks. Yours, on the other hand, will be new, untested, and could easily have serious bottlenecks if you haven’t profiled it extensively. It’s therefore very, very unlikely that you’ll be able to produce something more performant than the existing solutions, unless those existing solutions are old, barely maintained ones.

The only time this might be worthwhile is if all the existing implementations have boatloads of functionality, and you only need a small portion of that functionality. Even then, you should consider if it’s worth your while for a tiny speed boost. Or if you want to write a new library for it, go ahead - just don’t kid yourself about it being for the sake of performance.

Smaller frameworks are faster because they do less

Microframeworks such as Lumen are generally faster (at least in the artificial world of benchmarks), but that’s because they leave out functionality that’s not necessary for their targeted use case. Lumen is aimed at building microservices, and it leaves out things like templating, file handling, and other functionality not focused solely on building microservices. That means it’s less useful for other use cases. Any code that gets added to the application will make it marginally slower just by virtue of being there.

Under these circumstances it’s blindingly obvious that the framework that has to do less setup (eg instantiate fewer services, perform less operations on the request and response), is nearly always going to respond faster, regardless of suitability for more complex work.

If you start building a site with Lumen, but then discover that you need some functionality that Laravel has and Lumen doesn’t, you have two choices:

• Switch to Laravel
• Add that functionality to your application (either through additional packages or rolling it yourself)

I’ve often had plans to use Lumen for a project in the past, but then discovered that it would benefit from some of Laravel’s functionality. Under those circumstances I’ve switched straight over to Laravel - my time is too valuable to my employer to waste reimplementing functionality Laravel already has, and that functionality will inevitably have some overhead. Put it this way - I do a lot of Phonegap work, so building APIs is a big part of what I do, but I’ve only ever finished one project using Lumen (a push notification microservice). Every other time, sooner or later I’ve run into a situation where the additional functionality of Laravel would be useful and switched over.

There are occasions when a lighter framework like Lumen makes sense, but only when I simply don’t need the additional functionality of Laravel. It just doesn’t make sense to go for Lumen and then start adding functionality Laravel already has - any new implementation isn’t likely to be as solid, well-tested and performant as Laravel’s implementation.

Framework performance is often less relevant if you’re using Varnish

In my experience, if you have a site or API that is under heavy load, then if it’s possible to use Varnish with it, that will have a far more significant effect on performance than switching between PHP frameworks.

Because Varnish sits in front of your web server, when you’re serving cached content, anything after Varnish is completely irrelevant to the performance- it won’t hit the backend again until the cached content has expired. Varnish is effectively a key-value store, and is written in C, so it’s far more performant than just about any backend in any framework you could possibly write. And it’s configurable enough that with sufficient experience it can usually be helpful for most applications.

Varnish isn’t appropriate for every use case, and it doesn’t help with uncached requests (except by reducing the load on the application) but where high performance is necessary it can be a very big help indeed. The speed boost from having Varnish in front of your site and properly configured dwarfs any boost of a few milliseconds from switching PHP framework.

There are other HTTP caching servers available too - for instance, it’s possible to use Nginx as a web cache, and Cloudflare is a hosted service that offers similar performance benefits. Regardless, the same applies - if you can handle a request using the caching server rather than the application behind it, the performance will be immensely better, without having to change your application code.

ORM vs raw queries is a drop in the ocean

There will always be some overhead from using any ORM. However, this is nearly always so minor as to be a non-issue.

For example, while there might be some slight performance increase from writing raw SQL instead of using an ORM, it’s generally dwarfed by the cost of making the query in the first place. You can get a far, far bigger improvement in performance by efficiently caching the responses than by rewriting ORM queries in raw SQL.

An ORM does make certain types of slow inefficient queries more likely, as well as making “hidden” queries (such as in Laravel when it fetches the user from the session), but that’s something that can be resolved by using a profiler like Clockwork to identify the slow or unnecessary queries and refactoring them. Most ORM’s have tools to handle things like the N+1 problem - for instance, Eloquent has the with() method to eager-load related tables, which is generally a lot more convenient than explicitly writing a query to do the eager-loading for you.

Using an ORM also comes with significant benefits to developers:

• It’s generally easier to express relations between tables
• It helps avoid the mental context switch between PHP and SQL
• It does a lot of the work of sanitizing data for you
• It helps make your application portable between different databases (eg so you can run your tests using an in-memory SQLite database but use MySQL in production)
• Where you have logic that can’t be expressed using the ORM, it’s generally easy to drop down to writing raw SQL for that part

In my experience, querying the database is almost always the single biggest bottleneck (the only other thing that can be as bad is if you’re making requests to a slow third-party API), and any overhead from the ORM is a drop in the ocean in comparison. If you have a slow query in a web application, then rewriting it as a raw query is probably the very last thing you should consider doing, after:

• Refactoring the query or queries to be more efficient/remove unnecessary queries
• Making sure the appropriate indices are set on your database
• Caching the responses

Caching in particular is quite hard to do - it’s difficult to come up with a reliable and reusable strategy for caching responses without serving stale content, but once you can do so, it makes a huge difference to application performance.

Writing all your queries as raw queries is a micro-optimisation - it’s a lot of work for not that much payback, and it’s hardly ever worth the bother. Even if you have a single, utterly horrendous query or set of queries that has a huge overhead, there are better ways to deal with it - under those circumstances I’d be inclined to create a stored procedure in a migration and call that rather than making the query directly.

Summary

So to sum it up, if someone tells you you should use framework X because it’s faster than framework Y, they might be somewhat right, but that misses the point completely. Benchmarks are so artificial as to be almost useless for determining how your production code will perform. Any half-decent framework will give you the tools you need to optimise performance, and your use of those tools will have a far, far more signficant effect on the response time of your application than picking between different frameworks. I’ve never found a single MVC framework whose core is slow enough that I can’t make it fast enough with the capabilities provided.

Also, considering that these days server hardware is dirt cheap (at time of writing US$5 gets you a Digital Ocean droplet with 1GB of RAM for a month), whereas developers are far, far more expensive, it’s more cost effective to optimise for the developer’s time, not server time, so it makes sense to pick a framework that makes you productive, not one that makes the application productive. That’s no excuse for slow, shitty applications, but when all else fails, spinning up additional servers is a far more cost-effective solution than spending days on end rewriting your entire application in a different framework that benchmarks show might perform better by a few milliseconds.

A while back I provided details of the web server setup I used for Django applications. Nowadays I tend to use Laravel most of the time, so I thought I’d share an example of the sort of setup I use to deploy that.

Server OS

As before I generally prefer Debian Stable where possible. If that’s not possible for any reason then the current Ubuntu LTS is an acceptable substitute.

Web server

My usual web server these days is Nginx with PHP 7 or better via FPM. I generally use HTTP2 where possible, with SSL via Let’s Encrypt.

Here’s my typical Nginx config:

fastcgi_cache_path /etc/nginx/cache levels=1:2 keys_zone=my-app:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' https://placehold.it; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com ; font-src 'self' https://themes.googleusercontent.com; frame-src 'none'; object-src 'none'";
server_tokens off;
server {
    listen 80;
    listen [::]:80;
    server_name my-app.domain;
    return 301 https://$server_name$request_uri;
}
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    include snippets/ssl-my-app.domain.conf;
    include snippets/ssl-params.conf;
    client_max_body_size 50M;
    fastcgi_param HTTP_PROXY "";
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    root /var/www/my-app.domain/current/public;
    index index.php index.html index.htm;
    server_name my-app.domain;
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
    location ~ \.php$ {
        try_files $uri /index.php =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php/php7.0-fpm-my-app.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_cache my-app;
        fastcgi_cache_valid 200 60m;
    }
    location ~ /.well-known {
        allow all;
    }
    location ~* \.(?:manifest|appcache|html?|xml|json)$ {
        expires -1;
        gzip on;
        gzip_vary on;
        gzip_types application/json text/xml application/xml;
    }
    location ~* \.(?:rss|atom)$ {
        expires 1h;
        add_header Cache-Control "public";
        gzip on;
        gzip_vary on;
        gzip_types application/xml+rss;
    }
    location ~* \.(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
        expires 1M;
        access_log off;
        add_header Cache-Control "public";
    }
    location ~* \.(?:css|js)$ {
        expires 1y;
        access_log off;
        add_header Cache-Control "public";
        gzip on;
        gzip_vary on;
        gzip_types text/css application/javascript text/javascript;
    }
}

The times for FastCGI caching tend to vary in practice - sometimes it’s not appropriate to use it all, while for others it can be cached for some time.

It’s generally fairly safe to cache CSS and JS for a long time with a Laravel app if you’re using Mix to version those assets, so I feel comfortable caching them for a year. Images are a bit dicier, but still don’t change often so a month seems good enough.

I’ll typically give each application its own pool, which means copying the file at /etc/php/7.0/fpm/pool.d/www.conf to another file in the same directory, amending the pool name and path to set a new location for the socket, and then restarting Nginx and PHP-FPM. Here are the fields that should be changed:

; Start a new pool named 'www'.
; the variable $pool can be used in any directive and will be replaced by the
; pool name ('www' here)
[my-app.domain]
...
listen = /var/run/php/php7.0-fpm-my-app.sock

Database

I’m a fan of PostgreSQL - it’s stricter than MySQL/MariaDB, and has some very useful additional field types, so where possible I prefer to use it over MySQL or MariaDB.

Cache and session backend

Redis is my usual choice here - I make heavy use of cache tags so I need a backend for the cache that supports them, and Memcached doesn’t seem to have as much inertia as Redis these days. Neither needs much in the way of configuration, but you can get a slight speed boost by using phpiredis.

Queue

I sometimes use Redis for this too, but it can be problematic if you’re using Redis as the queue and broadcast backend, so these days I’m more likely to use Beanstalk and keep Redis for other stuff. I use Supervisor for running the queue worker, and this is an example of the sort of configuration I would use:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work --sleep=3 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/worker.log

This is fairly standard for Laravel applications.

Scheduler

I often make use of the Laravel scheduled tasks system. Here’s the typical cron job that would be used for that:

* * * * * php /var/www/artisan schedule:run >> /dev/null 2>&1

Again, this is standard for Laravel applications. It runs the scheduler every minute, and the scheduler then determines if it needs to do something.

Provisioning

To set all this up, I’ll generally use Ansible. In addition to this, I’ll generally also set up fail2ban to block various attacks via both HTTP and SSH.

Although Vagrant is very useful for ensuring consistency between development environments, it’s quite demanding on system resources. Running a virtual machine introduces quite a bit of overhead, and it can be troublesome to provision.

This week I was introduced to Lando as an alternative to Vagrant. Rather than running a virtual machine like Vagrant does by default, Lando instead spins up Docker containers for the services you need, meaning it has considerably less overhead than Vagrant. It also includes presets for a number of frameworks and CMS’s, including:

• Drupal 7
• Drupal 8
• Wordpress
• Laravel

Considering that Vagrant needs quite a bit of boilerplate to set up the server for different types of projects, this gives Lando an obvious advantage. The only issue I’ve had with it is that it’s been unreliable when I’ve had to use it on Windows, which I don’t do much anyway.

Getting started

Lando requires that you have Docker installed. Once that’s done you can download and install it fro the website. Then you can run lando init to set it up:

$ lando init
? What recipe do you want to use? wordpress
? Where is your webroot relative to the init destination? .
? What do you want to call this app? wp-site
NOW WE'RE COOKING WITH FIRE!!!
Your app has been initialized!
Go to the directory where your app was initialized and run
`lando start` to get rolling.
Check the LOCATION printed below if you are unsure where to go.
Here are some vitals:
 NAME      wp-site                                               
 LOCATION  /home/matthew/Projects/wp-site                        
 RECIPE    wordpress                                             
 DOCS      https://docs.devwithlando.io/tutorials/wordpress.html

Here I’ve chosen the wordpress recipe, in the current directory, with the name wp-site. This generates the following file as .lando.yml:

name: wp-site
recipe: wordpress
config:
  webroot: .

Then, if we run lando start, it will set up the required services:

$ lando start
landoproxyhyperion5000gandalfedition_proxy_1 is up-to-date
Creating network "wpsite_default" with the default driver
Creating volume "wpsite_appserver" with default driver
Creating volume "wpsite_data" with default driver
Creating volume "wpsite_data_database" with default driver
Creating wpsite_appserver_1 ... 
Creating wpsite_database_1 ... 
Creating wpsite_database_1
Creating wpsite_appserver_1 ... done
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4454k  100 4454k    0     0  3288k      0  0:00:01  0:00:01 --:--:-- 3290k
OS:     Linux 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64
Shell:  
PHP binary:     /usr/local/bin/php
PHP version:    7.1.13
php.ini used:   
WP-CLI root dir:        phar://wp-cli.phar
WP-CLI vendor dir:      phar://wp-cli.phar/vendor
WP_CLI phar path:       /tmp
WP-CLI packages dir:
WP-CLI global config:   
WP-CLI project config:  
WP-CLI version: 1.5.0
BOOMSHAKALAKA!!!
Your app has started up correctly.
Here are some vitals:
 APPSERVER URLS  https://localhost:32802
                 http://localhost:32803
                 http://wp-site.lndo.site
                 https://wp-site.lndo.site

Note the APPSERVER URLS section - the site can be accessed locally via HTTP or HTTPS. For this recipe, it also installs WP CLI.

If we run docker ps, we can see that it’s running three Docker containers:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                                               NAMES
2e920e152091        devwithlando/php:7.1-apache   "/lando-entrypoint.s…"   16 minutes ago      Up 16 minutes       0.0.0.0:32803->80/tcp, 0.0.0.0:32802->443/tcp                       wpsite_appserver_1
82ea60b1214f        mysql:latest                  "/lando-entrypoint.s…"   16 minutes ago      Up 16 minutes       0.0.0.0:32801->3306/tcp                                             wpsite_database_1
e51d831199d7        traefik:1.3-alpine            "/lando-entrypoint.s…"   About an hour ago   Up About an hour    0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:58086->8080/tcp   landoproxyhyperion5000gandalfedition_proxy_1

Apache lives in one container, MySQL in another, while the third runs Traefik, a lightweight load balancer, which listens on port 80. Traefik does the work of redirecting HTTP requests to the right place.

As I’ve been unhappy with the amount of resources Vagrant uses for a while, and I usually run Ubuntu (making using Docker straightforward), I’m planning on using Lando extensively in future. It’s lighter and faster to set up, and has sane defaults for most of the frameworks and CMS’s I use regularly, making it generally quicker and easier to work with.