Juan Treminio - Senior Web Developer Blog

OpenAPI Tutorial Part III: Paths and Basic Request Data

Sun, 06 Feb 2022 00:00:00 -0600

This is Part III of a multi-part series. Below are the links to other parts of this tutorial!

OpenAPI Tutorial Part I: Introduction to OpenAPI

OpenAPI Tutorial Part II: Common API Example

OpenAPI Tutorial Part III: Paths and Basic Request Data

The end result of this article can be found at jtreminio/openapi-tutorial branch “part-iii”. You can clone it by doing

$ git clone git@github.com:jtreminio/openapi-tutorial.git

Make sure to checkout branch part-iii.

OpenAPI Tutorial Part II: Common API Example introduced a basic API codebase that we will begin adding OpenAPI support to.

Today we will go over adding paths, and the basic types of request data.

Is Redoc Documentation Server Running?

Before you go any further, make sure your Redoc server is running:

$ ./bin/docs.sh Server started: http://127.0.0.1:8080 👀 Watching ./ for changes... 

Adding Paths

You will notice that the jtreminio/openapi-tutorial already contains some helpful comments to help developers know how each endpoint is supposed to work.

For example:

class PetController { /** * POST /pet */ public function petCreate(RequestInterface $request): string { // ... } } 

There are also endpoints using path parameters with these comments:

 /** * PUT /pet/{id}/photo */ public function petUpdatePhoto(RequestInterface $request): string { // ... } 

These comments are strictly for human readers, they do not affect your OpenAPI definitions. I included them because I expect many of your existing APIs might contain similar data. This might be the case for other parts of the codebase - you and your team have probably left behind comments to help each other better understand what the intended purpose of an endpoint is, how it is supposed to be called, what parameters it expects.

The benefit of using OpenAPI is you begin codifying these random and haphazard comments into a strict definition.

Definition of a Path

Paths are the endpoints your API makes available for users. They will contain the verb (GET, POST, PUT, DELETE, etc) and, if applicable, path parameters.

Paths combine the verb and the endpoint to create a unique definition. All this means is that POST /order is a different path than UPDATE /order.

Your First Path Definition

Let’s begin with PetController::petCreate(), which does not have any path parameters:

 /** * POST /pet */ public function petCreate(RequestInterface $request): string { // ... } 

Import the OpenApi\Annotations annotations namespace at the top of the class:

<?php declare(strict_types=1); namespace PetStoreApi\Controller; use OpenApi\Annotations as OA; use PetStoreApi\Model; use PetStoreApi\RequestInterface; 

Now, add our first path definition:

class PetController { /** * @OA\Post(path="/pet", * summary="Add a new pet", * operationId="petCreate", * tags={"Pet"}, * ) */ public function petCreate(RequestInterface $request): string { 

Finally, generate the new definitions file:

$ ./bin/generate.php Warning: @OA\Post() requires at least one @OA\Response() in \PetStoreApi\Controller\PetController->petCreate() in ./src/Controller/PetController.php on line 20 in ./vendor/zircote/swagger-php/src/Loggers/DefaultLogger.php on line 27 Wrote OAS file to ./bin/../openapi.yaml Running php-cs-fixer Loaded config default from "./.php-cs-fixer.php". Using cache file ".php-cs-fixer.cache". 

Ignore the warning for now, reload your Redoc browser window, and you should see the beginning of some documentation.


Your first endpoint definition

If you take a look at your openapi.yaml file you should see the change in the definition:

# ... paths: /pet: post: tags: - Pet summary: 'Add a new pet' operationId: petCreate 

Break It Down - Path Annotations

These four new lines of data are the core pieces of a path definition. Let’s go over each part before moving on.

Verb + Path

In our PetController::petCreate() we have added the following:

class PetController { /** * @OA\Post(path="/pet", * summary="Add a new pet", * operationId="petCreate", * tags={"Pet"}, * ) */ public function petCreate(RequestInterface $request): string { 

This is a path definition. zircote/swagger-php has an annotation for each major verb:

@OA\Delete()
@OA\Get()
@OA\Head()
@OA\Options()
@OA\Patch()
@OA\Post()
@OA\Put()
@OA\Trace()

You really only need @OA\Delete(), @OA\Get(), @OA\Post(), and @OA\Put().

The type of annotation you select and the path value will be combined in the generated definition.

@OA\Post(path="/pet") generates:

paths: /pet: post: # ... 

While @OA\Get(path="/pet") generates:

paths: /pet: get: # ... 

If you have two paths, one with @OA\Post(path="/pet") and the other with @OA\Get(path="/pet") you can clearly see how OpenAPI considers these separate definitions:

paths: /pet: get: # ... post: # ... 

Summary & Description

We also define a summary value, summary="Add a new pet". In Redoc summary is used for the left sidebar listing all endpoints available, along with the verb (Post). Each documentation generator will work differently and if you are using something else you will need to experiment yourself!

There is also description which we have not used yet. It is similar to summary except longer.

Operation ID

operationId is a unique identifier for this path, operationId="petCreate". It must be unique across your entire definitions file. The convention is up to you, some teams like to use {verb}{Path}{Action} like postPetCreate. Others use {action}{PathRoot} like createPet.

I use {pathRoot}{Action} like petCreate. It helps group common paths together:

petCreate
petGet
petUpdate
petDelete
petList
petFindBy
petUpdatePhoto

Adding a Body Definition

The POST /pet is expecting data as part of the body request, and a possible file upload.

parameter	type
`name`	string
`age`	integer
`type`	string `dog`, `cat`, or `fish`
`status`	string `available`, `pending`, or `sold`
`info`	object `DogInfo`, `CatInfo`, or `FishInfo`
`photo`	binary (file upload)

This endpoint does not use the id property in the Model\Pet class. We will get to that later.

Body Annotations

You tell the annotations library this is a body schema by using @OA\Schema() and defining the schema property:

/** * @OA\Schema(schema="Pet") */ class Pet { 

Add an @OA\Property() definition to the name property:

 /** * @var string * @OA\Property() */ public $name; 

The zircote/swagger-php annotation library supports auto-detecting the property type by reading the @var attached to each property, or detecting any default value assigned to the property. However, I find it is best to explicitly define the type:

 /** * @var string * @OA\Property(type="string") */ public $name; 

Now, if you generate the definitions file again ($ ./bin/generate.php) you will see the new schema:

components: schemas: Pet: properties: name: type: string type: object 

Do the same for age, type, and status:

 /** * @var int * @OA\Property(type="integer") */ public $age; // ... /** * @var string "dog"|"cat"|"fish" * @OA\Property(type="string") */ public $type; // ... /** * @var string "available"|"pending"|"sold" * @OA\Property(type="string") */ public $status = 'available'; 

So far the types are simple, type="integer" and type="string". What would you expect a file upload to be? If you guessed type="file" or type="binary" you would be forgiven for being wrong.

 /** * @var \SplFileInfo|null * @OA\Property(type="string", * format="binary", * ) */ public $photo = null; 

For now, let’s skip over adding a definition for info. I promise we will come back to it in later parts of this series, but discriminators are a bit more advanced than what you are learning right now.

Regenerate your definitions file ($ ./bin/generate.php) and behold:

components: schemas: Pet: properties: name: type: string age: type: integer photo: type: string format: binary type: description: '"dog"|"cat"|"fish"' type: string status: description: '"available"|"pending"|"sold"' type: string type: object 

Break It Down - Body Annotations

zircote/swagger-php is helping us out quite a bit here. We first tell it that this entire class is a schema definition:

/** * @OA\Schema(schema="Pet") */ class Pet { 

That generates this YAML:

components: schemas: Pet: 

You can change schema to any unique name you want, it is completely separate from the actual class name. We only used the same name for class and schema to make this easier to learn.

With the above, zircote/swagger-php now knows that the class directly following the annotation will contain more annotations that should be added to the definition. That is why this annotation works:

 /** * @var string "dog"|"cat"|"fish" * @OA\Property(type="string") */ public $type; 

The @OA\Property() annotations works in a similar way to the @OA\Schema() annotations - it is attached to the property directly following the annotation. So, the following would not affect $info:

 /** * @var string "dog"|"cat"|"fish" * @OA\Property(type="string") */ public $type; /** * @var DogInfo|CatInfo|FishInfo */ public $info; 

You will also notice that any text added in the @var docblock is used as the parameter’s description. We can change this later but keep it for now.

 type: description: '"dog"|"cat"|"fish"' type: string 

If you reload your documentation page you will not actually see any of these changes! For that, we need to reference this new Pet schema from the POST /pet path.

Referencing a Schema

Back in PetController::petCreate(), update the path annotation to:

 /** * @OA\Post(path="/pet", * summary="Add a new pet", * operationId="petCreate", * tags={"Pet"}, * @OA\RequestBody( * @OA\MediaType(mediaType="multipart/form-data", * @OA\Schema(ref="#/components/schemas/Pet"), * ), * ), * ) */ public function petCreate(RequestInterface $request): string { 

Refresh your documentation page and see the magic:


Your first request body

Break It Down - RequestBody Annotations

I just introduced two new annotations to you, @OA\RequestBody() and @OA\MediaType().

For an endpoint that expects body data (for example, from forms) we use the @OA\RequestBody() annotation. This annotation should only be combined with @OA\Post() and @OA\Put() (and @OA\Patch() if you ever use it).

@OA\MediaType() defines the content-type this endpoint will accept. For any endpoint that will accept a file upload, you must use multipart/form-data. You can use anything here, but should probably stick with the most common:

application/json
application/xml
text/plain; charset=utf-8

You can see more examples in Swagger’s Media Type page.

So far what we have is

paths: /pet: post: tags: - Pet summary: 'Add a new pet' operationId: petCreate requestBody: content: multipart/form-data: # ... 

Finally, you see the powerful ref for the first time. ref deserves to have a large chunk of text explaining it, but for now let’s keep it simple.

At a Glance - `ref`

ref is actually $ref when written to the openapi.yaml file. However, our annotations library expects ref. Here is what it looks like:

paths: /pet: post: tags: - Pet summary: 'Add a new pet' operationId: petCreate requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Pet' 

We use this special tool to reference schema defined in other parts of our definitions file. The first parts are the location OpenAPI can find the schema: #/components/schemas.

The last part is the actual schema name, Pet.

This is powerful because it allows us to reuse schemas without having to copy and paste their definitions over and over.

Enum Property Types

So far we have only added simple property types to our definitions, those that are string, integer, or string::binary (file upload).

The string::binary type clues us in that we can have further sub-types. One such example are enums.

The Pet::$type property is currently defined as

 /** * @var string "dog"|"cat"|"fish" * @OA\Property(type="string") */ public $type; 

However, the "dog"|"cat"|"fish" part is only a docblock description, it does not actually apply to the OpenAPI definition (outside of description).

We can easily tell OpenAPI this is an enum by using string::enum:

 /** * @var string * @OA\Property(type="string", * enum={"dog", "cat", "fish"} * ) */ public $type; 

Now our definitions file knows this is an enum and will require that the string is only one of the allowed:

components: schemas: Pet: properties: # ... type: type: string enum: - dog - cat - fish type: object 

Do the same for status:

 /** * @var string * @OA\Property(type="string", * enum={"available", "pending", "sold"} * ) */ public $status = 'available'; 

Regenerate your descriptions file:

components: schemas: Pet: properties: # ... type: type: string enum: - dog - cat - fish status: type: string enum: - available - pending - sold type: object 

Now reload your Redoc browser window and we see something useful:


Enums example

But Wait, There’s More

So far we have used zircote/swagger-php for things that we can easily do by manually-creating the definitions file. Let me show you one immediate benefit!

Enums are just arrays of values and are defined as you would expect:

 /** * @var string * @OA\Property(type="string", * enum={"dog", "cat", "fish"} * ) */ public $type; 

Since the annotations live in a PHP class, we can benefit from actual PHP code in the annotations. If this enum is just an array, why not create a constant that contains the possible values?

/** * @OA\Schema(schema="Pet") */ class Pet { public const TYPE_ENUM = [ 'dog', 'cat', 'fish', ]; public const STATUS_ENUM = [ 'available', 'pending', 'sold', ]; // ... /** * @var string * @OA\Property(type="string", * enum=\PetStoreApi\Model\Pet::TYPE_ENUM * ) */ public $type; // ... /** * @var string * @OA\Property(type="string", * enum=\PetStoreApi\Model\Pet::STATUS_ENUM * ) */ public $status = 'available'; 

Note that you need to use the FQDN when reference a constant in this manner.

This is only the most basic of what you can do with constants. As we go further along in this series we will see more useful examples!

Wrapping It Up

We have our first path defined! It even has some (basic) parameters, including a file upload and two enums!

In the next part of this series we will create more paths and their parameters, and we will go over how to add complex parameter types like arrays, objects, and arrays of objects.

So much left to do, but having the Redoc documentation to see our changes as we make them is a great motivator. I encourage you to continue playing with the API and reading both the Swagger OpenAPI Guide and the zircote/swagger-php documentation to further familiarize yourself with the concepts we have learned so far, and what we will learn in the upcoming parts of this series.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

OpenAPI Tutorial Part II: Common API Example

Sat, 05 Feb 2022 00:00:00 -0600

This is Part II of a multi-part series. Below are the links to other parts of this tutorial!

OpenAPI Tutorial Part I: Introduction to OpenAPI

OpenAPI Tutorial Part II: Common API Example

OpenAPI Tutorial Part III: Paths and Basic Request Data

In the first part of this series we

lightly touched on the differences between Swagger and OpenAPI
mentioned some things we can do with an OpenAPI definitions file
recommended tools for
- definitions file generation
- documentation generation
- SDK generation
decided to implement an API similar to the common Pet Store API
created annotations for generating an almost empty definitions file
viewed our empty documentation via Redoc

Today, I want to begin with an already created, simple API. The purpose of this tutorial series is not to show you how to build an API from scratch, but rather to learn how you can take your existing API and use OpenAPI to make the whole process better.

jtreminio/openapi-tutorial

I want to introduce you to my repo, jtreminio/openapi-tutorial. I will use this repo to follow along with the progress made in this tutorial series.

This repo contains a (very) basic API example and I highly suggest you clone it locally to follow along. For this article we will begin with branch part-ii.

$ git clone git@github.com:jtreminio/openapi-tutorial.git $ cd openapi-tutorial $ git checkout part-ii 

Let us take a look at what is going on with this API!

Controllers

We will have three basic controllers, one each for managing pets, customers, and orders. You have probably seen code like this before, it is a basic CRUD web application! Each controller will have the most common functionality:

Create
Retrieve
Update
Delete

Controllers represent data being sent from the user to our API. OpenAPI calls these requests.

Please note that this is a basic implementation of only the API structure. We are not going to implement database functionality like retrieving or persisting data. Anywhere that you see // persist $pet here or // find pet in DB here assume that we would have the logic for performing those actions there. However, for the purposes of this tutorial series we can just hand-wave those concerns away!

Pet Controller

The PetController class is used for managing pets. Our Pet Store will have an inventory of pets, and these endpoints are what will be used. Users can add new pets, update existing pets (including uploading a photo of a pet), search for pets by a unique id value or by some terms, and delete pets from the store.

Here is the list of actions and a short description for each:

endpoint	description
`POST /pet`	Add a new pet to the store
`GET /pet/{id}`	Gets a single pet by `{id}`
`GET /pet/findBy`	Finds one or more pets by `name`, `type`, or `status`
`GET /pet/list`	Returns paginated list of pets
`PUT /pet/{id}`	Update a single pet by `{id}`
`PUT /pet/{id}/photo`	Update a single pet’s photo
`DELETE /pet/{id}`	Delete a single pet by `{id}`

Customer Controller

The CustomerController class is used for managing customers. The actions available will be very similar to those found in the PetController.

Here is the list of actions and a short description for each:

endpoint	description
`POST /customer`	Add a new customer to the store
`GET /customer/{id}`	Gets a single customer by `{id}`
`GET /customer/findBy`	Finds one or more customers by `name`, `email_address`, `address`, or `phone_number`
`GET /customer/list`	Returns paginated list of customers
`PUT /customer/{id}`	Update a single customer by `{id}`
`DELETE /customer/{id}`	Delete a single customer by `{id}`
`PUT /customer/{id}/phone`	Add a phone number to an existing customer
`DELETE /customer/{id}/phone`	Delete a phone number from an existing customer

Order Controller

The OrderController class is used for managing orders. The actions available will be very similar to those found in the PetController.

Here is the list of actions and a short description for each:

endpoint	description
`POST /order`	Add a new order to the store
`GET /order/{id}`	Gets a single order by `{id}`
`GET /order/findBy`	Finds one or more orders by `customer_id`, `pet_id`, or `status`
`GET /order/list`	Returns paginated list of orders
`PUT /order/{id}`	Update a single order by `{id}`
`DELETE /order/{id}`	Delete a single order by `{id}`

Models

We have three root models, and a few extra models that will be nested within the root models.

These models contain no logic, they are strictly for holding data in a known shape (in other words, a DTO).

Pet Model

The Pet Model has some interesting properties:

property	description
`photo`	A photo of a pet. It is `nullable`. The photo comes from a user upload
`type`	An enum with an allowed value of `dog`, `cat`, or `fish`
`info`	Depending on the `type` value above, a different sub-class will be used. `DogInfo` for `dog`, `CatInfo` for `cat`, `FishInfo` for `fish`
`status`	Another enum with an allowed value of `available`, `pending`, or `sold`

Customer Model

The Customer Model has some interesting properties:

property	description
`address`	Address of the customer, represented by the `Address` model
`phone_numbers`	An array of `Phone` models

Order Model

The Order Model is pretty boring, with only one note-worthy property:

property	description
`status`	An enum with an allowed value of `placed`, `shipped`, or `delivered`

Why This API?

So the above API is pretty boring, right? Nothing exciting about it, at all. That is the point. Start off with something you have seen a thousand times before and apply some new tooling to end up with an amazing new result.

Looks can be deceiving, though. Yes, this API is boring, but it actually has some very interesting properties I would like to point out.

Verbs

We saw the use of several REST verbs above:

POST /pet
PUT /pet/{id}
GET /pet/{id}
DELETE /pet/{id}

There are a few more, but these four will meet almost all your needs. Their purpose is pretty clear and different from the other:

POST - Create a new record, like creating a new pet
PUT - Similar to POST but used to update en existing record, like changing the name of an existing pet
GET - Get an existing record, like searching for an existing pet
DELETE - Delete an existing record, like deleting an existing pet

Enums

If you are a PHP developer you know that Enums are available as of version 8.1.

However, you can define properties as enum in OpenAPI easily. The endpoint GET /customer/findBy has a type string property that can only be one of the following values:

name
email_address
address
phone_number

We will learn how to define this using OpenAPI.

Nested Objects

The Customer Model’s address property is a single non-nullable instance of the Address model.

Arrays

The Customer Model’s phone_numbers property is defined as an array of Phone objects, (Phone[]).

Polymorphism

Feels like you are back in college, right?

The Pet Model has a very special info property that can be one of three types. The value of this info property completely depends on the value of the type property:

scenario	`info` instance type
`type === "dog"`	`info` is an instance of `DogInfo`
`type === "cat"`	`info` is an instance of `CatInfo`
`type === "fish"`	`info` is an instance of `FishInfo`

In OpenAPI this is called a discriminator. Its purpose is to figure out the value of a property or properties, depending on the value of another property.

We will take a deeper look at discriminators as we go further along.

Wrapping It Up

We now have our starter API. It currently has no OpenAPI functionality, other than the setup we performed in OpenAPI Tutorial Part I: Introduction to OpenAPI. However, it provides a solid foundation that should be very familiar to most web developers, and focuses on what I believe to be the most common use-case for OpenAPI - working with an existing API codebase.

In the next part of this series we will begin adding OpenAPI to our API using the zircote/swagger-php annotations library. We should begin seeing both the power of OpenAPI and the benefits of using annotations for writing our OpenAPI definitions.

We still have a ways to go and much to learn!

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

OpenAPI Tutorial Part I: Introduction to OpenAPI

Mon, 31 Jan 2022 00:00:00 -0600

This is Part I of a multi-part series. Below are the links to other parts of this tutorial!

OpenAPI Tutorial Part I: Introduction to OpenAPI

OpenAPI Tutorial Part II: Common API Example

OpenAPI Tutorial Part III: Paths and Basic Request Data

The end result of this article can be found at jtreminio/openapi-tutorial branch “part-i”. You can clone it by doing

$ git clone git@github.com:jtreminio/openapi-tutorial.git

Make sure to checkout branch part-i.

Stop me if one of these scenarios sounds familiar:

You have a public API but no SDKS. You expect your clients to use cURL to use your API.
You have an internal API, but no documentation. Your team’s engineers add functionality to endpoints willy-nilly and now it is hard to keep track of what parameters, and the types of each parameter, each endpoint expects.
You have a product, but no API. You have no idea how to even create an API, documentation, or examples. How do I even REST?
You have a mature public API and SDKs across several different languages.
- Nancy, who maintained the Ruby SDK, left 2 years ago and no one else on your team knows how to code in Ruby so it is far behind the Python SDK.
- Joe maintains the Python SDK, but he is lazy and slow and does not add new features in a timely manner. The Python SDK has some features the Ruby SDK is missing but has not been updated in several months and now customers are opening GitHub issues demanding an update to at least apply a security patch to some random library it uses.
- Your team is mostly a PHP team so the PHP SDK is kind of, more or less, almost, but not quite, on par with the actual API implementation. cURL will always have access to all the latest API endpoints and parameters. Why can’t your customers just use cURL?
- LOL there is no Java or .Net SDK. We’re a web development company, not a Windows shop!

You can use OpenAPI to drastically reduce, and sometimes completely eliminate, the workload required for creating, maintaining, and updating your API definition, keeping public documentation on-par with your internal API representation, and generating, maintaining, and updating your SDKs.

From a single YAML (or JSON) file you can define cURL examples, callback events, OAuth, and so much more. To be clear, while most of the work required to get these benefits will be done for you once you have your OpenAPI definitions file ready, some (especially SDK generation) will still require fine-tuning to both get around bugs in the tooling and to fit your specific use-case.

OpenAPI? I Thought It Was Swagger?

Swagger’s initial release was in 2011. The company that invented Swagger was eventually purchased by SmartBear Software. In 2016 the Swagger specification was renamed to OpenAPI Specification.

Now, “Swagger” refers to the tools offered by the SmartBear Software company, including:

They offer several more tools, and if you want to read much more details on the differences and products offered by Swagger, read their What Is the Difference Between Swagger and OpenAPI? page.

“OpenAPI” refers only to the actual specification. In other words, OpenAPI is what is defined in these files.

I will refer back to Swagger’s tools only when comparing them to other tools that, in my opinion, offer more robust support for the latest OpenAPI features.

The distinction here is important. Swagger is a group of tools, but OpenAPI is just a spec. By itself, OpenAPI does not actually do anything. For that, we must rely on the overall ecosystem built around the spec. This will become clearer as you read along.

What Can I Do With OpenAPI?

At the risk of being too simplistic and getting corrected by internet nerds the world over, the OpenAPI Spec (OAS) allows you to define a RESTful API in a YAML or JSON file(s), which we will call your definitions file(s).

You can:

define endpoints (POST /account/create)
what parameters the endpoints will accept (maybe the POST /account/create endpoint expects an email_address)
responses from the API to the requester in the form of
- HTTP codes (HTTP 200 OK)
- or full JSON responses
authorization, like Basic Auth, API keys, OAuth2, etc

There are a few more main points in the OAS and we will get to them in due time. The above all exist in your definitions file, but it is just a static text file. The file itself just implements the rules from the OAS. It is once you have your spec file that you can begin playing around with the tooling that has evolved alongside the OAS.

All of the following tools do their work from your definitions file, that is where the magic of OpenAPI begins to shine.

Documentation

Several tools exist that read your definitions file and generate beautiful documentation for end-users to read.

The documentation generated can look different based on which generator you use. For example, some will create beautiful HTML pages you can open in your browser. Others can spit out Markdown (or other) files.

There are a number of products that you can read about here. Some are open-sourced, others are paid. Each one generates pages that look different from the others, and each one implements the OAS differently as well!

SDK Generation

You can generate SDKs for end-users in a number of different languages. There are two major open-sourced generators:

Additionally, I know of one major commercial product called APIMatic that can generate SDKs, documentation, and several others things. They might be a good option if you want a third-party company to handle almost everything OpenAPI-related for you, but are somewhat expensive.

Before Getting Started, Tools to Use

The OpenAPI tool ecosystem is large and fast paced. Because the OAS is just a file that does not do anything by itself, figuring out all the different possibilities and deciding which tools to use can be daunting.

Rather than showing your OpenAPI definitions code and letting you figure out what to do with it, I figure it is easier to recommend some tools you can install on your machine before we dive in.

Generating Your Definitions File

You can write your definitions file by hand but I have found that to be cumbersome and slow. While the OAS allows reusing definitions so you do not have to copy/paste the same things over and over, it is fairly limited.

For generating your definitions file I recommend using an annotations-based tool. While this series can be considered language-agnostic, meaning whether you are strongest in PHP, Javascript, Ruby, etc, I will use a PHP-based tool called zircote/swagger-php to help generate the OpenAPI definitions file from PHP annotations.

There are a number of similar projects for other languages, and the information in this series should translate easily! Here are a few:

PHP - zircote/swagger-php
Typescript - tsoa
Python - drf-yasg
Ruby - source2swagger
See more languages

These types of annotations-based tools offer several benefits over manually maintaining your definitions file:

They are based on your language of choice and allow you to use language features like constants, greatly reducing the need to copy/paste data where the OAS’ reusability falls short.
Annotations can be placed alongside the code they are tied to. If you have a class that lists all properties an endpoint expects you can add property annotations in the same place. This makes maintaining your actual implementation with your OpenAPI definitions much simpler.
Your IDE may assist you with auto-complete

To help reduce language barriers I will first show code using the zircote/swagger-php annotations and then show the result of generating the definitions so you can compare between your results and mine.

You can install zircote/swagger-php via composer with composer require zircote/swagger-php. I will provide setup instructions later on.

If you are not using PHP please follow the instructions for your specific tool.

Generating Documentation

For documentation generation I highly recommend using redoc-cli. It is a CLI-based tool that reads your definitions file and starts a server on your local machine, using Redoc.

Redoc is a free, MIT-licensed tool, and the company that created it also offers a paid, commercial product called Redocly.

I am not receiving any kickbacks for recommending Redoc, I simply have tested a complex definitions file and found that Redoc is the only one that supports all of the OAS 3.0’s features. As we come across those specific features in this tutorial series I will point them out.

Install redoc-cli by following the instructions in their quickstart guide.

You can install it globally by running npm i -g redoc-cli.

SDK Generation

For SDK generation, stick with openapi-generator. It has a large, active community and the number of languages it supports is quite impressive.

After extensive testing I have found openapi-generator to be the least buggy and most feature-complete. That said, each language that openapi-generator supports is maintained by separate contributors, so code quality, features, and usability will vary greatly between them!

Follow their instructions to install.

If you are on MacOS I highly recommend installing via Homebrew: brew install openapi-generator.

If you are on Linux (or have Docker installed on Windows), use Docker.

openapi-generator is a Java application and its dependencies might be a bit frustrating to setup.

Choosing an API to Implement

For this tutorial series we will create an API similar (but different) to the Pet Store API from the ground up. Doing so will help make it clear how all the separate pieces fit together to create the final product. I will try to make the process mimic how a company with an existing, manually-maintained API would implement OpenAPI.

Additionally, we will extend this API to include some more advanced features available in OAS 3.x. Not all APIs are as simple as the default Pet Store example, so I am hoping this will turn out to be a good learning experience.

We will make mistakes, and implement code that may look weird or seem like alot of copy and pasting, but I want this tutorial series to seem organic. Not everyone begins with a well-made API before moving to OpenAPI, and I want to make this useful as a real-world guide.

Initial Setup

Assuming you have already installed the tools from above we can now create the scripts to call these tools. First, our project’s basic directory structure.

Organizing Our Code

We will create a directory to contain our executables so we do not have to memorize the commands for regenerating the definitions file, starting the documentation server, and (eventually) creating our SDKs. This will be the bin directory.

Create directories to hold our Controller and Model classes:

$ mkdir -p ./bin ./src/Controller ./src/Model $ tree . ├── bin │ └── // ... empty └── src ├── Controller │ └── // ... empty └── Model └── // ... empty 

Setting Up `zircote/swagger-php`

Install it with composer:

$ composer require zircote/swagger-php Using version ^4.2 for zircote/swagger-php ./composer.json has been created Running composer update zircote/swagger-php Loading composer repositories with package information # ... snip Package operations: 10 installs, 0 updates, 0 removals - Installing symfony/yaml (v5.4.3): Extracting archive - Installing zircote/swagger-php (4.2.4): Extracting archive 1 package suggestions were added by new dependencies, use `composer suggest` to see details. Generating autoload files 

Create a file at ./bin/generate.php:

#!/usr/bin/env php <?php require_once __DIR__.'/../vendor/autoload.php'; use OpenApi\Generator; use Symfony\Component\Yaml\Yaml; $openapi = Generator::scan([__DIR__.'/../src']); $yaml = $openapi->toYaml( Yaml::DUMP_OBJECT_AS_MAP ^ Yaml::DUMP_EMPTY_ARRAY_AS_SEQUENCE ^ Yaml::DUMP_MULTI_LINE_LITERAL_BLOCK ); $oas_file = __DIR__.'/../openapi.yaml'; file_put_contents($oas_file, $yaml); echo "Wrote OAS file to {$oas_file}\n"; 

You can make it executable with chmod +x ./bin/generate.php or run it directly with php.

All we have done here is told zircote/swagger-php (namespaced as OpenApi) to scan the ./src directory for its annotations, then write the generated definitions to the ./openapi.yaml file.

Whenever we make a change to our OpenAPI annotations we can run this script to regenerate our definitions file.

Setting Up `redoc-cli`

Create a file at ./bin/docs.sh:

#!/usr/bin/env bash set -e DIR=$(cd `dirname $0` && pwd) redoc-cli serve \ "${DIR}/../openapi.yaml" \ --port=8080 \ --watch \ --options.requiredPropsFirst=1 

Make it executable with chmod +x ./bin/docs.sh.

Once we create our initial definitions file we can run this script to keep Redoc open in a browser. Whenever we regenerate the definitions file refreshing the browser will show the latest changes.

Create OpenApi Entry File

Before we can run the generator for the first time we need to create a file with the minimum annotations requirements. Create a ./src/OpenApi.php file with the following contents:

<?php declare(strict_types=1); namespace PetStoreApi; use OpenApi\Annotations as OA; /** * @OA\OpenApi(openapi="3.0.3") * @OA\Info(title="Pet Store Api", * version="1.0.0", * ) * @OA\Server(url="https://petstore.example.com/") */ class OpenApi { } 

The @OA annotations will be read by the zircote/swagger-php library when generating our definitions file.

If you are following along in PHP you may want to read a little of the zircote/swagger-php documentation. If you are using another language, do the same with your tool!

Initial Generation

When you run the generator script it will create a file for you. Do not worry about warnings at this point, we will fill out the requirements in the next steps.

$ php ./bin/generate.php Warning: Required @OA\PathItem() not found in ./vendor/zircote/swagger-php/src/Loggers/DefaultLogger.php on line 27 Wrote OAS file to ./bin/../openapi.yaml 

Take a look at the newly-created ./openapi.yaml file. It should have the following contents:

openapi: 3.0.0 info: title: 'Pet Store Api' version: 1.0.0 servers: - url: 'https://petstore.example.com/' 

Now you can run the documentation script to get Redoc working!

$ ./bin/docs.sh Server started: http://127.0.0.1:8080 👀 Watching ./ for changes... 

Open http://127.0.0.1:8080 in your browser and you should see the Redoc documentation. It will be bare bones right now, but we will quickly start filling this in.

Final Project Structure

As of now you should have a project structure that looks like this:

$ tree -a --filesfirst . ├── composer.json ├── composer.lock ├── openapi.yaml ├── bin │ ├── docs.sh │ └── generate.php ├── src │ ├── OpenApi.php ├── Controller │ └── // ... empty └── Model └── // ... empty └── vendor ├── autoload.php ├── bin │ └── openapi -> ../zircote/swagger-php/bin/openapi └── // ... snip 

Extra Credit

We are going to be writing a lot of annotations, and sometimes an IDE may get confused with exactly how many spaces to indent multi-nested definitions.

For example, the info block in your definitions file can contain much more information, like so:

/** * @OA\OpenApi(openapi="3.0.3") * @OA\Info(title="Pet Store Api", * version="1.0.0", * @OA\License(name="MIT", * url="https://opensource.org/licenses/MIT" * ), * ) */ 

If you care about consistent spacing the above should look like this:

/** * @OA\OpenApi(openapi="3.0.3") * @OA\Info(title="Pet Store Api", * version="1.0.0", * @OA\License(name="MIT", * url="https://opensource.org/licenses/MIT" * ), * ) */ 

It is a small thing, but once you get 4, 5 levels deep the small differences can get annoying.

Let’s use friendsofphp/php-cs-fixer to automatically format our code every time we regenerate our definitions file:

$ composer require --dev friendsofphp/php-cs-fixer Using version ^3.4 for friendsofphp/php-cs-fixer ./composer.json has been updated # ... snip 

Create a ./.php-cs-fixer.php file to contain the configuration options:

<?php $finder = PhpCsFixer\Finder::create() ->in(__DIR__.'/src'); $config = new PhpCsFixer\Config(); return $config->setRules([ '@DoctrineAnnotation' => true, '@Symfony' => true, ]) ->setFinder($finder) ->setRiskyAllowed(true); 

Now update the ./bin/generate.php file so its contents are as follows:

#!/usr/bin/env php <?php require_once __DIR__.'/../vendor/autoload.php'; use OpenApi\Generator; use Symfony\Component\Yaml\Yaml; $openapi = Generator::scan([__DIR__.'/../src']); $yaml = $openapi->toYaml( Yaml::DUMP_OBJECT_AS_MAP ^ Yaml::DUMP_EMPTY_ARRAY_AS_SEQUENCE ^ Yaml::DUMP_MULTI_LINE_LITERAL_BLOCK ); $oas_file = __DIR__.'/../openapi.yaml'; file_put_contents($oas_file, $yaml); echo "Wrote OAS file to {$oas_file}\n"; echo "\nRunning php-cs-fixer\n"; shell_exec(__DIR__.'/../vendor/bin/php-cs-fixer fix'); 

Your annotations will be fixed every time you generate:

$ ./bin/generate.php Warning: Required @OA\PathItem() not found in ./vendor/zircote/swagger-php/src/Loggers/DefaultLogger.php on line 27 Wrote OAS file to ./bin/../openapi.yaml Running php-cs-fixer Loaded config default from "./.php-cs-fixer.php". Using cache file ".php-cs-fixer.cache". 

You should have ended up with this:

$ tree -a --filesfirst . ├── .php-cs-fixer.cache ├── .php-cs-fixer.php ├── composer.json ├── composer.lock ├── openapi.yaml ├── bin │ ├── docs.sh │ └── generate.php ├── src │ ├── OpenApi.php ├── Controller │ └── // ... empty └── Model └── // ... empty └── vendor ├── autoload.php ├── bin │ ├── openapi -> ../zircote/swagger-php/bin/openapi │ └── php-cs-fixer -> ../friendsofphp/php-cs-fixer/php-cs-fixer └── // ... snip 

Wrapping It Up

Today we briefly spoke about the differences between Swagger and OpenAPI. We touched on the various things you can do with an OpenAPI definitions file, and some of the tools you can use to do these things.

We have installed tools to generate our definitions file, run a documentation server, and (optionally) do some formatting of our annotations declarations.

We even generated our first definitions file (even though it is missing some required data) and saw our documentation page (even though it’s pretty blank!).

In the next part of this series we will start implementing actual endpoints and their parameters. Eventually we will begin adding more advanced features like authentication.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

All-in-One PHP-FPM + Nginx/Apache Containers

Tue, 18 Dec 2018 00:00:00 -0600

In a previous post, Docker PHP/PHP-FPM Configuration via Environment Variables, I described how to use environment variables to configure PHP FPM and CLI. With my method you can override one of over 650 PHP INI settings. More INI settings are but a PR away!

In a more recent post, PHP Modules Toggled via Environment Variables I extended this concept to include modules. You can now easily (and quickly!) enable over 20 common modules, including GD, Imagick, Mongodb, and more! The images I built come with over 30 modules installed, with a small selection enabled by default, and the rest toggable by simple environment variables.

With these two concepts combined my other post, Developing at Full Speed with Xdebug becomes that much more powerful, because now you can easily enable Xdebug via a very simple env var flag. A truly delay-free debug-ready development environment has never been easier to maintain.

However, my Xdebug post uses a method that requires two separate PHP-FPM servers, one with Xdebug enabled and another without. To run a normal PHP project you would be required to run 4 containers in total:

2 PHP (FPM)
1 Nginx (Webserver)
1 MySQL (Database)

While this falls inline with Docker’s vision of one-container per responsibility, it is a hassle to have to maintain separate containers for both Nginx and PHP, with the main reason being that both separate services need access to your project’s source code in order to properly work.

You can argue that PHP-FPM depends on a webserver to actually function and this responsibility can be considered as one. Thus, one container running both Nginx (or your webserver of choice) and PHP is desirable.

Managing Multiple Services Under Docker

Docker containers can run multiple services just fine. However, the daemon actively checks only a single main process which includes monitoring it for health (is this service still running?) and using it for stdout/stderr logging.

The official Docker documentation recommends using either writing your own wrapper script, or using a process manager like supervisord.

You could install Supervisord. It is a great process manager! However, why go that route when you can use a Docker-ready image to build from?

Enter baseimage-docker.

This Ubuntu 18.04-based image comes preinstalled and configured with nifty tools like runit for process management (Does not require crazy weird init.d config files!), syslog-ng for log management, and cron which is sorely-missing from most Docker images!

I will not go into detail about baseimage-docker as their documentation is top-notch. Later you will be able to see what I created using this tool as an example.

Setting up Nginx

The Nginx images extend my PHP images. We already have a strong PHP-focused base to start from and can simply add Nginx to it.

The runit file is fairly simple.

In it we spin up Nginx as most folks do: nginx -g "daemon off;"

However, the Nginx build I am using comes with support for the handy Perl module. This allows us to pass environment variables to the Nginx config files. Normally Nginx has no support for this (which is a huge gaping hole, in my opinion) but using the Perl module you get a workable implementation.

The one thing I have baked in to the Nginx images is passing in an Xdebug port.

This works nicely with my Developing at Full Speed with Xdebug blog post.

You can setup Nginx env var as follows:

perl_set $phpfpm_xdebug_port 'sub { return $ENV{"PHPFPM_XDEBUG_PORT"}; }'; 

The runit file then looks like this:

PHPFPM_XDEBUG=${PHPFPM_XDEBUG:-""} if [ "${PHPFPM_XDEBUG,,}" = "on" ]; then PHPFPM_XDEBUG_PORT=9999 else PHPFPM_XDEBUG_PORT=9000 fi exec nginx -g "daemon off; env PHPFPM_XDEBUG_PORT=${PHPFPM_XDEBUG_PORT};" 

Everything hinges on the PHPFPM_XDEBUG env var. If it is set to On or on Nginx is told to use port 9999 for Xdebug requests, otherwise to use the standard 9000.

The final piece of the Nginx puzzle is the virtual host config:

map $cookie_XDEBUG_SESSION $my_fastcgi_pass { default 127.0.0.1:9000; xdebug 127.0.0.1:${phpfpm_xdebug_port}; } 

If Nginx detects a cookie named XDEBUG_SESSION it uses ${phpfpm_xdebug_port}, otherwise defaults to the standard 9000.

What happens when Xdebug support is not requested is that both cases would be interpreted as simply

map $cookie_XDEBUG_SESSION $my_fastcgi_pass { default 127.0.0.1:9000; xdebug 127.0.0.1:9000; } 

This allows us to toggle Xdebug without having to actually make any file changes.

Likewise, if Xdebug support is enabled (via the PHPFPM_XDEBUG=On env var) then it would be interpreted as

map $cookie_XDEBUG_SESSION $my_fastcgi_pass { default 127.0.0.1:9000; xdebug 127.0.0.1:9999; } 

Setting up PHP-FPM

The PHP-FPM runit init file is already created at the jtreminio/php:7.2 level. Nginx can immediately start working with this existing service and nothing else is required on your part. However, we want to also optionally add a second PHP-FPM instance with Xdebug enabled but only when PHPFPM_XDEBUG=On!

runit requires a bash file with +x flag set (making it executable). One route we could have taken was to simply mount a file volume via a docker-compose.yml file but unfortunately any files shared in this manner are not set as executable. This means we have to bake in the init file into the image itself. However, if we want to continue down the path of not requiring any file system changes and make everything configurable via env vars then we need to figure out a way to prevent runit from running a second PHP-FPM instance when Xdebug is not to be enabled.

The solution I implemented does just that.

If PHPFPM_XDEBUG is not set to On|on then a harmless, low-resource service is run:

1 exec tail -f /var/log/fpm-xdebug-tail

This is because runit needs to run something, and simply exiting at this point would cause the daemon to try restarting the service over and over. There is no disable flag to tell runit, “Hey, don’t run this!”.

Otherwise, if PHPFPM_XDEBUG=On then a second PHP-FPM service is run using

exec /usr/sbin/php-fpm \ -d FPM.pid="/var/run/php-fpm/php-fpm-xdebug.pid" \ -d FPM.listen="127.0.0.1:9999" \ [...] 

We override the pid and listen directives to prevent any collisions with the non-Xdebug PHP-FPM instance.

Bonus: Setting up Apache

Since we are using PHP-FPM the webserver engine becomes much easier to switch out. As long as it can speak TCP it is ok, which means adding Apache support is fairly simple.

The Dockerfile for the Apache image is very similar to the Nginx one.

Likewise, the runit init file for Apache looks similar and the runit init file for PHP-FPM is identical.

Apache has built-in support for env vars (looking at you, Nginx) so only the vhost config file needs changed.

Apache has no map directive, but does support If/Else:

<FilesMatch "\.php$"> <If "%{HTTP_COOKIE} =~ /XDEBUG_SESSION/"> SetHandler proxy:fcgi://127.0.0.1:${PHPFPM_XDEBUG_PORT} </If> <Else> SetHandler proxy:fcgi://127.0.0.1:9000 </Else> </FilesMatch> 

The same idea from Nginx applies here, $PHPFPM_XDEBUG_PORT defaults to 9000 unless PHPFPM_XDEBUG=On then it defaults to 9999.

Try It Out

You can try this out fairly quickly, but first some helpers.

Grab the PhpStorm bookmarklets. You do not need to use PhpStorm to use these, they simply create an XDEBUG_SESSION cookie in your browser. This is the cookie the webserver listens for. Again, if this cookie does not exist all traffic is routed to the non-Xdebug PHP-FPM instance, and vice-versa.

Next, setup Traefik. You may remember Traefik from my Traefik on Docker for Web Developers post. It is an amazing tool that helps map domain names to containers.

TRAEFIK=$(docker container ls --filter name=traefik_proxy | grep -c traefik_proxy || true) if [ ${TRAEFIK} -eq 0 ]; then NETWORK=$(docker network ls --filter name=traefik_webgateway | grep -c traefik_webgateway || true) if [ ${NETWORK} -eq 0 ]; then docker network create --driver bridge traefik_webgateway fi docker container run -d \ --name traefik_proxy \ --network traefik_webgateway \ --publish 80:80 \ --publish 8080:8080 \ --restart always \ --volume /var/run/docker.sock:/var/run/docker.sock \ --volume /dev/null:/traefik.toml \ traefik --api --docker --docker.domain=docker.localhost --logLevel=DEBUG fi 

We setup Traefik via bash instead of a docker-compose.yml file so it is not tied to a specific project (the container name would have the parent directory prepended to it otherwise).

Now you can run the PHP/Nginx instance using:

version: '3.2' networks: public: external: name: traefik_webgateway services: web: image: jtreminio/php-nginx:7.2 labels: - traefik.backend=php-nginx - traefik.docker.network=traefik_webgateway - traefik.frontend.rule=Host:php-nginx.localhost - traefik.port=8080 networks: - public volumes: - ./:/var/www/ environment: - PHP.display_errors=On - PHP.error_reporting=-1 - PHPFPM_XDEBUG=On 

Run docker-compose up and then open php-nginx.localhost in Chrome (or another browser if you have dnsmasq installed).

Create an index.php file that simply contains <?php phpinfo(); to test this out.

If you want to try with Apache, use the following:

version: '3.2' networks: public: external: name: traefik_webgateway services: web: image: jtreminio/php-apache:7.2 labels: - traefik.backend=php-apache - traefik.docker.network=traefik_webgateway - traefik.frontend.rule=Host:php-apache.localhost - traefik.port=8080 networks: - public volumes: - ./:/var/www/ environment: - PHP.display_errors=On - PHP.error_reporting=-1 - PHPFPM_XDEBUG=On 

Run docker-compose up and then open php-apache.localhost in Chrome (or another browser if you have dnsmasq installed).

You can try the Xdebug feature by either commenting out PHPFPM_XDEBUG=On or changing it to anything not On|on: PHPFPM_XDEBUG=Off.

Wrapping It Up

This project is the end result of weeks/months testing different solutions, always poking and refactoring while trying to achieve a truly immutable image.

Not to say that this project is done, but I believe it is on the right path to becoming something many developers and companies can build their Docker deployments on.

Not needing to manage separate containers for PHP/Nginx|Apache, enabling or disabling Xdebug and many other modules, and configuring PHP with nothing more than simple environment variables reduces complexity and speeds up development time.

I would love to hear your feedback on this and would be elated if you would drop me a line if you decide to implement my images into your workflow.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

PHP Modules Toggled via Environment Variables

Fri, 14 Dec 2018 00:00:00 -0600

In my previous post, Docker PHP/PHP-FPM Configuration via Environment Variables I introduced my new PHP image that uses environment variables to configure its INI settings.

The Dockerfile is a beast, to be sure. At over 650 lines it documents my attempt at bringing true flexibility to an immutable PHP image.

With it, you can set any number of PHP INI settings without having to rebuild your image:

$ docker container run --rm \ -e PHP.error_reporting=-1 \ jtreminio/php:7.2 php -i | \ grep error_reporting error_reporting => -1 => -1 

As part of my aim for immutability, I install all of the available modules in Ondřej Surý’s great PHP PPA.

This increases the final Docker image size by quite a bit, but in reality file size should not be in your list of top concerns. Ease of deployment, repeatability and speed of spinning up a new instance should top your list.

While my PHP image hit these three spot-on, it did introduce another issue: loading all the available PHP modules would slow the engine down, or introduce small incompatibilities and bugs to your code that you were not expecting.

If you are not doing any image work, do you really need to have the GD or Imagick modules enabled?

More clearly, why have the Redis or Mongodb modules enabled if you ever only work with MySQL?

Why Not Use Environment Variables To Load Modules?

The php.ini file included with my PHP images looks like this:

allow_url_fopen = ${PHP.allow_url_fopen} allow_url_include = ${PHP.allow_url_include} 

This works because I have pre-defined all the variables to be used as environment variables:

1 2 ENV PHP.allow_url_fopen=1 \ PHP.allow_url_include= \

If a user does not pass PHP.allow_url_fopen to the Docker image, it defaults to 1, and PHP.allow_url_include defaults to no value.

I am forced to pre-define values because some INI settings will crash the PHP engine if left empty. They must have a non-empty value.

Likewise, PHP modules are loaded as:

extension=redis.so 

Leaving this setting blank will immediately crash PHP. You cannot simply try to not load redis.so by leaving the setting as extension=.

If you decide to break immutability in your Docker images and delete the INI files that have these extension=* lines in them, you may run into other problems like having to set up an entry-point directive in your Dockerfile to delete these files on startup, or if you want your PHP-FPM service to have Redis disabled but may want to use it for PHP-CLI.

Removing the need to change the container’s filesystem was the main purpose behind going with configuration-by-environment-variables in the first place, and this implementation only accomplishes half that.

Meet `PHP_INI_SCAN_DIR`

When compiling PHP you tell it where to find the default php.ini file and you can also define where it should scan for any additional INI files.

In Ondřej Surý’s build the additional files path is set to /etc/php/7.2/cli/conf.d for CLI:

$ docker container run -it --rm \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: /etc/php/7.2/cli/conf.d Additional .ini files parsed: /etc/php/7.2/cli/conf.d/10-mysqlnd.ini, /etc/php/7.2/cli/conf.d/10-opcache.ini, # more lines below 

A special env var called PHP_INI_SCAN_DIR can be used to override this path.

By setting PHP_INI_SCAN_DIR to empty you can disable everything:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR= \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: (none) Additional .ini files parsed: (none) 

, or change the scan directory completely, whether the directory has INI files or not:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=/foo/bar \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: /foo/bar Additional .ini files parsed: (none) 

, and add additional scan directories to the default:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=:/foo/bar \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: :/foo/bar Additional .ini files parsed: /etc/php/7.2/cli/conf.d/10-mysqlnd.ini, /etc/php/7.2/cli/conf.d/10-opcache.ini, # more lines below 

Finally, PHP_INI_SCAN_DIR is powerful enough that you can even define multiple directories:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=:/foo/bar:/baz/bam \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: :/foo/bar:/baz/bam Additional .ini files parsed: /etc/php/7.2/cli/conf.d/10-mysqlnd.ini, /etc/php/7.2/cli/conf.d/10-opcache.ini, # more lines below 

One thing to keep in mind is that PHP_INI_SCAN_DIR only works with directories. There is not a PHP_INI_SCAN_FILE equivalent and you cannot use it to include specific files. It will load all INI files in a defined directory.

Using PHP_INI_SCAN_DIR you can introduce incredible flexibility to your environment by selectively loading INI files.

Loading Modules with `PHP_INI_SCAN_DIR`

Since PHP_INI_SCAN_DIR loads all INI files in a directory you have to create separate directories for all your optional modules.

If you install the Redis PHP module, an INI file is automatically created at /etc/php/7.2/cli/conf.d/20-redis.ini. This is convenient if you want to actually enable this module, but not desired when you are creating a Docker image with flexibility in mind!

Simply creating another directory within /conf.d to hold the INI file will not work, since PHP_INI_SCAN_DIR scans all children. For example, moving it /etc/php/7.2/cli/conf.d/redis/redis.ini will not change anything as it will still be loaded by default. You must move it outside of the scanned directory.

In my Docker image I chose /etc/php/extra-mods as the directory to hold all optional module’s INI files, so the Redis INI would go in /etc/php/extra-mods/redis/redis.ini.

This INI only contains a single line:

$ docker container run -it --rm \ jtreminio/php:7.2 cat /etc/php/extra-mods/redis/redis.ini extension=redis.so 

If you do not tell PHP to load this file, Redis is not enabled, but you can now easily enable it with PHP_INI_SCAN_DIR:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=:/etc/php/extra-mods/redis \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: :/etc/php/extra-mods/redis Additional .ini files parsed: /etc/php/7.2/cli/conf.d/10-mysqlnd.ini, [...snip...] /etc/php/extra-mods/redis/redis.ini 

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=:/etc/php/extra-mods/redis \ jtreminio/php:7.2 php -i | grep redis 655:Redis Support => enabled 656:Redis Version => 4.1.1 667:Registered save handlers => files user redis rediscluster 

My Docker image comes with many modules installed, and with this you can now enable exactly the ones that you want:

$ docker container run -it --rm \ -e PHP_INI_SCAN_DIR=:/etc/php/extra-mods/redis:/etc/php/extra-mods/gd:/etc/php/extra-mods/xdebug:/etc/php/extra-mods/mongodb \ jtreminio/php:7.2 php --ini Configuration File (php.ini) Path: /etc/php/7.2/cli Loaded Configuration File: /etc/php/7.2/cli/php.ini Scan for additional .ini files in: :/etc/php/extra-mods/redis:/etc/php/extra-mods/gd:/etc/php/extra-mods/xdebug:/etc/php/extra-mods/mongodb Additional .ini files parsed: /etc/php/7.2/cli/conf.d/10-mysqlnd.ini, [...snip...] /etc/php/extra-mods/redis/redis.ini, /etc/php/extra-mods/gd/gd.ini, /etc/php/extra-mods/xdebug/xdebug.ini, /etc/php/extra-mods/mongodb/mongodb.ini 

This is extremely powerful. Enabling modules and changing INI settings can be done solely through environment variables, without needing to rebuild your Docker image!

One small nitpick: this line is extremely long and unwieldy:

PHP_INI_SCAN_DIR=:/etc/php/extra-mods/redis:/etc/php/extra-mods/gd:/etc/php/extra-mods/xdebug:/etc/php/extra-mods/mongodb 

My images conveniently symlinks the much shorter /p to /etc/php/extra-mods, so the above now becomes:

1 PHP_INI_SCAN_DIR=:/p/redis:/p/gd:/p/xdebug:/p/mongodb

Bonus: Separate FPM and CLI `PHP_INI_SCAN_DIR` Values

I hope you can already see the possibilities, but let me mention one that may not be immediately obvious.

You can define PHP_INI_SCAN_DIR for PHP-FPM, while also setting it to a different value for PHP-CLI.

$ docker container run -it --rm \ --name testing \ -e PHP_INI_SCAN_DIR=:/p/redis \ jtreminio/php:7.2 *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh... *** Running /etc/my_init.d/10_syslog-ng.init... Dec 15 05:24:09 a9e40e3240c2 syslog-ng[13]: syslog-ng starting up; version='3.13.2' *** Booting runit daemon... *** Runit started as PID 22 Dec 15 05:24:10 a9e40e3240c2 cron[27]: (CRON) INFO (pidfile fd = 3) Dec 15 05:24:10 a9e40e3240c2 cron[27]: (CRON) INFO (Running @reboot jobs) [15-Dec-2018 05:24:10] NOTICE: fpm is running, pid 32 [15-Dec-2018 05:24:10] NOTICE: ready to handle connections [15-Dec-2018 05:24:10] NOTICE: systemd monitor interval set to 10000ms 

PHP-FPM is now running with Redis enabled. Running any PHP scripts within the container will also load Redis, even if not through PHP-FPM:

$ docker container exec \ testing php -i | grep redis 10:Scan this dir for additional .ini files => :/p/redis 49:/p/redis/redis.ini 653:redis 655:Redis Support => enabled 656:Redis Version => 4.1.1 667:Registered save handlers => files user redis rediscluster 877:PHP_INI_SCAN_DIR => :/p/redis 1485:$_SERVER['PHP_INI_SCAN_DIR'] => :/p/redis 2101:This program is free software; you can redistribute it and/or modify 

However, you can also run PHP-CLI with Redis disabled, without having any effect on the running PHP-FPM instance:

$ docker container exec \ -e PHP_INI_SCAN_DIR=/etc/php/7.2/cli/conf.d \ testing php -i | grep redis 2094:This program is free software; you can redistribute it and/or modify 

About the Image Size…

Like I mentioned, my Docker images are not optimized for size. The benefits far outweigh something as cheap as file size.

Surprisingly, when you compare my images with the official images you will notices that there is not much difference in size!

$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE jtreminio/php 7.2 9f0102616848 26 hours ago 381MB php 7.2-fpm 2bd622691e6e 4 days ago 371MB 

10MB larger size with the benefit of not needing to compile modules, nor having to maintain a separate php.ini file and instead being able to do all this through simple environment variables!

A worthy trade-off, in my opinion.

Want to Try It Out?

My images are all available on the Docker hub.

I maintain versions for PHP 7.3 - 5.6.

You can see all the Dockerfiles used to generate the images in my repo.

Wrapping It Up

I would like to thank Ondřej Surý for his years-long work in maintaining amazing PHP, Nginx, Apache (and more!) repos.

I would also like to thank Derick Rethans for letting me bounce ideas off of him. While my final solution may seem easy and obvious in hind-sight, actually arriving at this point required hours of Derick being supernaturally patient with me while I massacre his PHP language.

The resulting Docker images accomplish just about everything I had in mind. As I created, tinkered and optimized them through months of use at work, and asked the coworkers on my team to also use them.

I hope you may find them as useful and easy to use as we have, and your productivity to match!

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

Docker PHP/PHP-FPM Configuration via Environment Variables

Thu, 15 Nov 2018 00:00:00 -0600

ed: If you want to run the containers right now, jump ahead to How to Use the Images.

For several months now I have been working on PuPHPet.com’s replacement. It is a Docker-based GUI functionally similar to PuPHPet.

Docker, like Vagrant, allows sharing directories and files from the host to the container/VM. Unlike Vagrant, Docker images are easy to create, easy to share and easy to configure.

One of the most convenient differences is that with Docker containers you can pass flags to the container which can be used for configuration (if the image was created with this in mind).

For example, with the MariaDB image you create a new container and define the database credentials:

docker run -it --rm \ -e MYSQL_DATABASE=dbname \ -e MYSQL_USER=dbuser \ -e MYSQL_PASSWORD=dbpassword \ mariadb 

This creates a new MariaDB container with a database named dbname, user dbuser and password dbpassword. You do not need to create a separate configuration file, the above takes care of that for you!

What if we could do the same for a PHP container? Having to keep track of a separate INI file for both PHP and PHP-FPM is not nearly as smooth as how MariaDB’s image can be configured.

Our Goals

I will talk you through the thought process required for creating a Docker image that is capable of configuring its service using environment variable flags:

-e {ENV_NAME}={ENV_VALUE} 

I will detail the gotchas you would encounter trying to create this for yourself from scratch, and the solutions I came up with and implemented.

Quick Introduction to Docker Images

It will definitely help if you know Docker at a usable level. If you know Docker enough to only run pre-built images then I hope this will help solidify your understanding of how Docker works.

That said I am not an expert in the internal workings of Docker, and I might get some things wrong.

Docker images are blueprints for Docker containers. You need an image to run one or more instances which are known as containers.

An image can be grabbed from the Docker Hub or you can easily create your own images by writing a Dockerfile. This is a simple text file that contains instructions for creating your image.

Unless you are doing some hardcore stuff, you always start with an existing base image. For example, FROM ubuntu:18.04 means we will start with an Ubuntu 18.04 image to create our final image.

You can have as many layers as you want, with each new image extending the parent and adding more functionality. That is exactly what we will end up doing today.

PHP INI with Environment Variables (env vars)

A little-known fact is that PHP’s INI file (and PHP-FPM conf, too!) can be configured normally,

display_errors=0 error_reporting=Off date.timezone=UTC 

but it can also read env vars!

display_errors=${DISPLAY_ERRORS} error_reporting=${ERROR_REPORTING} date.timezone=${DATE_TIMEZONE} 

PHP-FPM can also be configured this way:

pm = ${FPM_PM} pm.max_children = ${FPM_PM_MAX_CHILDREN} pm.start_servers = ${FPM_PM_START_SERVICE} 

Since we can pass environment variable values when spinning up a Docker container, this can be combined to remove the need for maintaining a separate php.ini file in your project’s repo!

Caveats

A few things to keep in mind:

You need to be explicit about which settings you want to allow env vars for. This means you can only use env vars if you have set the INI value to read from an env var (display_errors=${DISPLAY_ERRORS})!
If an env var is not set, the value will be empty. This means you must either set a default value when creating the initial INI file (not always desired), or the INI value must allow empty values.
Bash has a default value fallback when working with variables. It looks like FOO=${BAR:-"default value"}. While this would have been great for our use case, INI files are not processed as Bash and thus the default value trick does not work here.

The first point means you cannot use just any Docker image, it must have been built with env vars in mind from the start. A quick search on the Docker Hub does not bring up many results for this - just about all major PHP Docker images require managing the PHP INI file directly.

The second point hurts us a little bit because some INI settings simply cannot have empty values. For example, you cannot have an extension=. PHP will complain.

In FPM if you have slowlog= FPM will crash immediately. Unfortunately some INI settings are not amenable to env vars unless you enforce a value for them.

The final point is a minor inconvenience. You cannot define a setting with an inline env var in the same line. You have to do this process separately.

This means you cannot simply have your INI file look like:

display_errors=${DISPLAY_ERRORS:-"Off"} error_reporting=${ERROR_REPORTING:-0} date.timezone=${DATE_TIMEZONE:-"UTC"} 

This would have been great but since it is strictly a Bashism it does not work within the PHP INI parser.

This forces you to keep a separate list of env vars and their values, and a separate list of INI settings calling the env vars.

We will go through these points in more detail later on.

Building Docker Image with Env Vars

Adding an env var to a Docker image within a Dockerfile is simple enough:

FROM ubuntu:18.04 ENV allow_url_fopen=1 

Now all containers that use this image and all images that extend this image, would have the env var allow_url_fopen available to them.

To change this value when spinning up a container you would simply do

docker container run -it --rm \ -e allow_url_fopen=0 \ {image_name} 

and in the php.ini using

allow_url_fopen=${allow_url_fopen} 

would allow us to change the setting on the fly without having to further edit a static INI file.

My end goal was not to have just one or two settings available for changing, I want developers to take my images and plug them into their projects and configure as much as possible only through env vars. This means I needed to add as many INI settings as I can.

This leads to a problem. Dockerfile cannot read and grab env vars from a separate file. You have to define all env vars in the Dockerfile to be able to use them in child images or containers.

In a docker-compose file you can simply use env_file like so:

services: my_special_service: image: something env_file: - env-file.env 

If you are spinning up a container from an existing image you can likewise do

docker container run \ --env-file=${PWD}/env-file.env \ [..] 

However, you cannot do the same when building a new image. There is no ENV_FILE in a Dockerfile - you have to list each env var one by one!

The problem is more obvious when I tell you I identified over 650 PHP INI settings I wanted to set as env vars. Then realize I want to support all active versions of PHP (5.6, 7.0, 7.1, 7.2, 7.3) and that means each of the four Dockerfiles now need to have 650+ lines just for the env vars.

If Dockerfile supports an ENV_VAR directive this would be a single-line solution. Since it does not, I needed to come up with some other way to add these hundreds of env vars, without junking up my Dockerfiles.

Create a Parent Image

The solution turned out to be quite simple! Since each Dockerfile extends a parent image (via FROM), and everything available to that base image is available to the current Dockerfile, I created a Dockerfile that does nothing but define all the INI env vars my PHP images will be able to use.

You can see it here.

For settings that have no default value, and can remain blank, I did

1 PHP.allow_url_include=

For settings that ship with default values, I did

PHP.allow_url_fopen=1 

Notice that I namespaced all my variables. The actual variable name is PHP.allow_url_fopen and so the INI file must reference this as allow_url_fopen=${PHP.allow_url_fopen}. This simply helps avoid naming collisions. You might also know that in Bash all variable names must only contain alphanumeric or _ characters. The . in PHP.allow_url_fopen makes this an invalid variable name in Bash.

Thankfully we are not reading these values in Bash, we are reading them in a PHP INI/FPM config file so the variable names are perfectly valid.

Child Images

Now that I have created an image that defines all the INI settings I want to expose to env vars, I can create child images that can reference the env vars in their php.ini files!

You can see the INI file that all my images inject here.

Any image that extends the previous image can use the defined env vars.

The following slice of php.ini:

allow_url_fopen = ${PHP.allow_url_fopen} allow_url_include = ${PHP.allow_url_include} always_populate_raw_post_data = ${PHP.always_populate_raw_post_data} 

is read as the following by the PHP engine:

allow_url_fopen = 1 allow_url_include = always_populate_raw_post_data = 

Blank values are perfectly acceptable for these settings.

Likewise, in the PHP-FPM config file we have:

pm = ${FPM.pm} pm.max_children = ${FPM.pm.max_children} pm.start_servers = ${FPM.pm.start_servers} 

which is read as

pm = dynamic pm.max_children = 5 pm.start_servers = 2 

I deliberately chose not to add all possible settings. Like I mentioned earlier, FPM’s slowlog setting must have a value. If it is blank FPM immediately dies with a segfault. Settings like this where developers may not always want to have but cannot leave blank are not friendly to the env var method.

Removing CMD from Dockerfile

One thing I ran into was that the service defined via Dockerfile’s CMD will not receive updated env var values!

For example, in my PHP images I would have PHP-FPM service handled via CMD and this service would pick up env var values as set at that point. If the Dockerfile had ENV FPM.pm=dynamic that is what the CMD service would use, even if you later changed the env var value.

The list of possible solutions included having the CMD reload the current env vars, or starting a new session, or any number of hacky things just to get around this limitation.

What I ended up doing was simply removing CMD from my Dockerfiles. Now the PHP images are truly open between being used for CLI or FPM! You simply call the PHP-FPM service when spinning up the container.

How to Use the Images

It is simple enough to use these images.

For the default INI values:

$ docker container run --rm \ jtreminio/php:7.2 php -i | grep error_reporting 195:error_reporting => 0 => 0 

To override the defaults:

$ docker container run --rm \ -e PHP.error_reporting=-1 \ jtreminio/php:7.2 php -i | grep error_reporting 195:error_reporting => -1 => -1 

You can just as easily override multiple settings:

$ docker container run --rm \ -e PHP.error_reporting=-1 \ -e PHP.display_errors=On \ -e "PHP.date.timezone=America/Chicago" \ jtreminio/php:7.2 php -i | \ egrep 'error_reporting|display_errors|date.timezone' display_errors => STDOUT => STDOUT error_reporting => -1 => -1 date.timezone => America/Chicago => America/Chicago 

To run PHP-FPM you just call the service:

$ docker container run --rm \ -e PHP.error_reporting=-1 \ -e PHP.display_errors=On \ -e "PHP.date.timezone=America/Chicago" \ jtreminio/php:7.2 /usr/bin/php-fpm [15-Nov-2018 23:32:39] NOTICE: fpm is running, pid 9 [15-Nov-2018 23:32:39] NOTICE: ready to handle connections [15-Nov-2018 23:32:39] NOTICE: systemd monitor interval set to 10000ms 

You can test if PHP-FPM is reading env vars as well by doing:

$ docker container run --rm \ -e PHP.error_reporting=-1 \ -e PHP.display_errors=On \ -e "PHP.date.timezone=America/Chicago" \ -e FPM.pm=foobar \ jtreminio/php:7.2 /usr/bin/php-fpm [15-Nov-2018 23:33:08] ERROR: [/etc/php/7.2/fpm/php-fpm.conf:14] unable to parse value for entry 'pm': invalid process manager (static, dynamic or ondemand) [15-Nov-2018 23:33:08] ERROR: failed to load configuration file '/etc/php/7.2/fpm/php-fpm.conf' [15-Nov-2018 23:33:08] ERROR: FPM initialization failed 

Wrapping It Up

Not all services are amenable to configuration by environment variables, which is a shame. Tools like Nginx require special modules (Lua) for this functionality.

The PHP core team was forward-thinking enough to have added support for env vars long ago, and we get to enjoy its benefits.

Some future nice-to-haves would include default values, but for now this works pretty well.

Having to manage less files to run your Docker containers is a win in my book.

If you see some INI settings that you think should be included, please submit a PR to the repo.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

Setting Up a Static Site with Hugo and Push to Deploy

Wed, 08 Aug 2018 00:00:00 -0500

For several years this blog was generated using the PHP static site generator Sculpin. I switched to Grav before deciding it was not for me. My blog does not contain dynamic data that requires PHP processing, and static HTML with JS is fine.

One of the issues I had with Grav was its requirement of both a PHP-FPM and Nginx/Apache service to properly serve content.

After researching available options, I decided to switch to the amazing Hugo.

Hugo has several benefits over other generators:

completely static output is generated from Markdown/HTML files,
a single Go binary with no outside dependencies, or a container can be used to generate static files,
tons of themes,
only requires a webserver for production deployment (Nginx/Apache)

Our goals

Today I will walk you through the complete process of setting up a static website that you can deploy new versions with a simple git push.

Pushing to your repo will trigger an automated build process that will generate minified HTML/CSS/JS files, package them in an Nginx image, add a new image to Docker Hub, deploy a new container on your host and automatically generate and maintain free SSL certificates using Let’s Encrypt.

The process outlined here is what I have created and use for this blog, jtreminio.com. Each new build and deployment currently takes around a few minutes and is completely automated.

The only prerequisite is you have Docker installed on your local machine. All other dependencies will be in Docker containers.

Technology and services used

We will configure and run several technologies, including:

Hugo for static site generator,
Ansible for configuring the server,
Docker for containers,
Traefik for routing traffic to correct container, and automatic SSL certificates,
Watchtower for keeping latest Docker image running on your site.
Let’s Encrypt for free, automated SSL certificate.

Everything that is used is completely free and open sourced, other than the host. If you are in need of a host I can recommend Digital Ocean ¹. The basic $5/month plan will be more than enough. If you have your own host you would prefer to use, by all means use it!

Configure server

First we need to install Docker, Traefik and Watchtower on the server.

I have created a simple Ansible-based configuration that

installs Docker,
opens ports 80, 443 and 8080 on firewall (optional),
adds Traefik and configures Let’s Encrypt support,
creates a Watchtower container.

The only things you must configure are all handled by creating a .env file and filling out the following:

name	description
SERVER_IP	IP address of server
SSH_USER	SSH username, “root” on Ubuntu
SSH_PRIVATE_KEY	Path to SSH private key on your machine
LE_EMAIL	Email to use for Let’s Encrypt

Run Ansible

You can start Ansible by running ./init in the root of the repo directory.

It will create an Ansible container on your local machine that will connect to and configure your defined remote servers.

The local Ansible container is removed once it finishes running.

If all goes well you should see something similar to

PLAY RECAP ********************************************** remote : ok=8 changed=8 unreachable=0 failed=0 

The important part is failed=0.

If you go to your blog’s URL you will see an invalid certificate warning.

This is fine! We have not actually created the blog container and Traefik has not generated an SSL certificate for it yet.

Setting up Hugo

Next we will get Hugo running on our local machine.

I have created a Hugo bootstrap repo that comes with some tools already added.

All you need to do is clone two repos:

git clone https://github.com/jtreminio/hugoBasicExample.git cd hugoBasicExample git clone https://github.com/nanxiaobei/hugo-paper.git \ themes/hugo-paper 

And you can start the Hugo server:

1 ./bin/hugo-server

Now open http://localhost:1313/ and you will see a fully working Hugo blog.

Feel free to explore Hugo in more detail by visiting gohugo.io.

Manual deployment process steps

With a single command Hugo takes all your HTML/Markdown content and generates static files in /public.

You can see this process by running

1 ./bin/hugo-publish

You will see your Markdown posts in HTML files, nested within directories that match your blog structure. Hugo also copies over all CSS/JS/etc files that are in your root /static or the theme’s.

You could take all this static content and deploy to production as-is, but we can run some minify tools to get the file sizes down.

Hugo does no post-processing and everything must be done by third-party tools. I have added a minify script that you can run with:

1 ./bin/minify

It takes all HTML, CSS and JS files and minifies them down to a much smaller size.

Finally, you can run an Nginx container to make sure your site looks properly.

This local Nginx container will be exactly the same as what you deploy to production:

1 ./bin/nginx

All these steps can be run manually, but that is a waste of time. Better to automate the process!

Docker multi-stage builds

Our end goal is to automate the 3 steps above and end up with a single, tiny image we can deploy to our production server (automatically).

Docker recently came out with multi-stage builds.

It means you can create a single Dockerfile with as many sequential stages as you need to generate a single, final image.

I have created a Dockerfile which takes the 3 steps above and runs through them. If you create the image on your computer you will end up with a single, tiny container at the end:

$ docker image build -t hugo-test . Sending build context to Docker daemon 1.383MB Step 1/17 : FROM alpine/git ---> 1e76d5809b62 Step 2/17 : COPY . /data ---> a473877e4ad9 Step 3/17 : WORKDIR /data ---> Running in 6e1b6e2796a4 Removing intermediate container 6e1b6e2796a4 ---> 1fcaafec077f Step 4/17 : RUN rm -rf themes/* ---> Running in 020d0c4f303f Removing intermediate container 020d0c4f303f ---> 00a81909f7a0 Step 5/17 : RUN git clone https://github.com/nanxiaobei/hugo-paper.git themes/hugo-paper ---> Running in 4d7c71cd51ac Cloning into 'themes/hugo-paper'... Removing intermediate container 4d7c71cd51ac ---> 5e67ef78f4b8 Step 6/17 : FROM skyscrapers/hugo:0.46 ---> 434ff241d9e8 Step 7/17 : COPY --from=0 /data /data ---> 3d27347872c5 Step 8/17 : WORKDIR /data ---> Running in f0875071a444 Removing intermediate container f0875071a444 ---> ca8120476886 Step 9/17 : RUN hugo ---> Running in e2b6817fe000 | EN +------------------+----+ Pages | 12 Paginator pages | 0 Non-page files | 0 Static files | 2 Processed images | 0 Aliases | 0 Sitemaps | 1 Cleaned | 0 Total in 15 ms Removing intermediate container e2b6817fe000 ---> cc2be3328f07 Step 10/17 : FROM mysocialobservations/docker-tdewolff-minify ---> 43c3688d88ad Step 11/17 : COPY --from=1 /data/public /data/public ---> 144634f56841 Step 12/17 : WORKDIR /data ---> Running in 404cb0f24509 Removing intermediate container 404cb0f24509 ---> d0a02742aa3c Step 13/17 : RUN minify --recursive --verbose --match=\.*.js$ --type=js --output public/ public/ ---> Running in 38f21d784856 INFO: use mimetype text/javascript INFO: expanding directory public/ recursively INFO: minify input file public/js/custom.js INFO: minify to output directory public/ INFO: ( 68.167µs, 32 B, 49.2%, 954 kB/s) - public/js/custom.js INFO: 3.055423ms total Removing intermediate container 38f21d784856 ---> d0d80a7d1ab1 Step 14/17 : RUN minify --recursive --verbose --match=\.*.css$ --type=css --output public/ public/ ---> Running in 81e9840eb053 INFO: use mimetype text/css INFO: expanding directory public/ recursively INFO: minify input file public/css/style.css INFO: minify to output directory public/ INFO: (389.209µs, 6.0 kB, 100.0%, 15 MB/s) - public/css/style.css INFO: 3.797968ms total Removing intermediate container 81e9840eb053 ---> 80108c675341 Step 15/17 : RUN minify --recursive --verbose --match=\.*.html$ --type=html --output public/ public/ ---> Running in d0c1c70b3e80 INFO: use mimetype text/html INFO: expanding directory public/ recursively INFO: minify 30 input files INFO: minify to output directory public/ INFO: (283.292µs, 2.2 kB, 99.7%, 7.6 MB/s) - public/404.html INFO: (192.584µs, 3.4 kB, 99.8%, 18 MB/s) - public/about/index.html INFO: (286.917µs, 3.7 kB, 99.8%, 13 MB/s) - public/categories/development/index.html INFO: ( 68µs, 275 B, 100.0%, 4.0 MB/s) - public/categories/development/page/1/index.html INFO: (219.375µs, 3.7 kB, 99.8%, 17 MB/s) - public/categories/golang/index.html INFO: ( 56.709µs, 260 B, 100.0%, 4.6 MB/s) - public/categories/golang/page/1/index.html INFO: (253.918µs, 2.7 kB, 99.8%, 11 MB/s) - public/categories/index.html INFO: ( 56.625µs, 239 B, 100.0%, 4.2 MB/s) - public/categories/page/1/index.html INFO: (272.709µs, 5.8 kB, 99.9%, 21 MB/s) - public/index.html INFO: ( 68.126µs, 206 B, 100.0%, 3.0 MB/s) - public/page/1/index.html INFO: (1.140128ms, 56 kB, 100.0%, 49 MB/s) - public/post/creating-a-new-theme/index.html INFO: (487.084µs, 15 kB, 100.0%, 30 MB/s) - public/post/goisforlovers/index.html INFO: (317.792µs, 5.8 kB, 99.9%, 18 MB/s) - public/post/hugoisforlovers/index.html INFO: (252.542µs, 5.2 kB, 99.9%, 21 MB/s) - public/post/index.html INFO: (349.251µs, 11 kB, 99.9%, 31 MB/s) - public/post/migrate-from-jekyll/index.html INFO: ( 77µs, 221 B, 100.0%, 2.9 MB/s) - public/post/page/1/index.html INFO: (317.834µs, 3.8 kB, 99.8%, 12 MB/s) - public/tags/development/index.html INFO: ( 80.792µs, 257 B, 100.0%, 3.2 MB/s) - public/tags/development/page/1/index.html INFO: (351.584µs, 3.8 kB, 99.8%, 11 MB/s) - public/tags/go/index.html INFO: ( 68.083µs, 230 B, 100.0%, 3.4 MB/s) - public/tags/go/page/1/index.html INFO: (280.542µs, 3.8 kB, 99.8%, 14 MB/s) - public/tags/golang/index.html INFO: ( 69.042µs, 242 B, 100.0%, 3.5 MB/s) - public/tags/golang/page/1/index.html INFO: (255.334µs, 3.0 kB, 99.8%, 12 MB/s) - public/tags/hugo/index.html INFO: ( 68.417µs, 236 B, 100.0%, 3.4 MB/s) - public/tags/hugo/page/1/index.html INFO: (221.125µs, 3.7 kB, 99.8%, 17 MB/s) - public/tags/index.html INFO: ( 81.125µs, 221 B, 100.0%, 2.7 MB/s) - public/tags/page/1/index.html INFO: (198.083µs, 2.9 kB, 99.8%, 15 MB/s) - public/tags/templates/index.html INFO: (118.167µs, 251 B, 100.0%, 2.1 MB/s) - public/tags/templates/page/1/index.html INFO: ( 242µs, 2.9 kB, 99.8%, 12 MB/s) - public/tags/themes/index.html INFO: ( 67.75µs, 242 B, 100.0%, 3.6 MB/s) - public/tags/themes/page/1/index.html INFO: 112.866806ms total Removing intermediate container d0c1c70b3e80 ---> f31a796feec7 Step 16/17 : FROM nginx:alpine ---> 36f3464a2197 Step 17/17 : COPY --from=2 /data/public /usr/share/nginx/html ---> 5c0ffab7f6be Successfully built 5c0ffab7f6be Successfully tagged hugo-test:latest 

Inside this single Dockerfile are 4 FROM sections. What Docker actually ends up doing is creating 3 intermediary images, and one final image. The final image contains nothing but what you explicitly COPY into it, and the end result is a tiny image:

$ docker image ls REPOSITORY <... snip ...> SIZE <none> <... snip ...> 262MB hugo-test <... snip ...> 18.8MB <none> <... snip ...> 106MB <none> <... snip ...> 25.1MB 

This tiny image is what we end up deploying to production. It contains Nginx and all the static, minified files.

You can test it yourself by running:

docker container run --rm -it -p 8080:80 hugo-test 

and going to http://localhost:8080

Docker Hub Automated Builds

After you have played around a bit with Hugo, commit any changes you have made and push to a public repo on Github.

In our next steps we will get the Docker Hub to do the exact same process as above whenever we push a new change to Github.

If you do not yet have an account, create one at hub.docker.com.

We are now going to grant the Docker Hub to access our Github repos and add hooks.

This simply means whenever a commit is pushed to the repo, Github will notify the Docker Hub and it will automatically create a new image for us.

Go to Linked Accounts & Services and follow the directions.

Next, go to the Automated Builds page and click Create Auto-build Github.

From there you can find the repo you created earlier.

There are currently two bugs with the Docker Hub GUI when creating an automated build.

The repository you create for the automated build must not exist on Docker Hub. For example, my Hub username is jtreminio and my repo’s name is jtreminio.com (found here). Using the GUI found here the Hub will auto-populate the fields for you, even if you already have a repo by that name! Either change the name on this page or delete your existing repo. This is on the Docker Hub, not on Github!

The URL you end up in, after the GUI found here, may be incorrect! For me it generated a URL that ended with /github/form/jtreminio/jtreminio.com/?namespace=github. This silently fails when you submit the form. The ?namespace= part should actually contain your Docker Hub username! I had to change my URL to /github/form/jtreminio/jtreminio.com/?namespace=jtreminio

After you follow the instructions you will find the Docker Hub repo page now includes several more options than before, including Dockerfile, Build Details, and Build Settings.

If you go to Build Settings you can manually start your first build by clicking Trigger on the right side of the page. This may take a few minutes.

Starting your blog

Once the first build is finished on the Docker Hub we can create the initial container for our blog on our server.

SSH into your server and run the following:

NAME=jtreminio_com HOST=jtreminio.com IMAGE="jtreminio/jtreminio.com" docker container run -d --name ${NAME} \ --label traefik.backend=${NAME} \ --label traefik.docker.network=traefik_webgateway \ --label traefik.frontend.rule=Host:${HOST} \ --label traefik.port=80 \ --label com.centurylinklabs.watchtower.enable=true \ --network traefik_webgateway \ --restart always \ ${IMAGE} 

Make sure to change NAME, HOST and IMAGE to your own information!

A few things will now happen:

The container with your website inside will start,
Traefik detects this new container and automatically generates a new, free SSL certificate from Let’s Encrypt. It will continue monitoring this certificate and renew it long before it expires, all without you needing to worry about it.
Watchtower takes note of this new container, but does nothing right now.

If you go to your website URL you will see your blog up and running with a brand new SSL certificate!

Watchtower

So what exactly does Watchtower do? If you run

1 docker container logs watchtower

you may not see anything very interesting at first. The magic happens when you make changes to your website, commit and push to Github, and after the Docker Hub automatically creates a new image of your website.

Watchtower polls the Docker Hub every few minutes to detect if any of the containers you are currently running have new image versions. Once Docker Hub finishes creating the new image with the latest changes of your website, Watchtower will automatically download the image, gracefully shut down your blog container and immediately restart it with the new image, and your new changes.

Here is what the logs show when this happens:

root@docker:/opt# docker container logs -f watchtower time="2018-08-09T00:28:50Z" level=info msg="First run: 2018-08-09 00:33:50 +0000 UTC" // ... time="2018-08-09T00:33:53Z" level=info msg="Found new jtreminio/jtreminio.com:latest image (sha256:5a8c9299091b6892753128792a6d6c90f26dd27ed10c5286b3fc8f0b8799c503)" time="2018-08-09T00:33:57Z" level=info msg="Stopping /jtreminio_com (ebae9539acfcedf2279115f2c19ebddaf3c34271aa5d048142c6b90d091bf987) with SIGTERM" time="2018-08-09T00:33:58Z" level=info msg="Creating /jtreminio_com" 

Watchtower can monitor any number of containers and is the final piece in our automated puzzle.

Wrapping up

Today you learned how to utilize free, open source tools to automate your blog deployment process.

While Docker Hub automated builds may not be suitable for more complex requirements, it can easily meet what we created today.

No more FTP, nor more pulling from repo directly from your server. Automating this boring and error-prone process helps lift a small weight off of your shoulders and lets you focus on what you enjoy doing best: writing about things you love.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

Affiliate link, help support this free blog! ↩

Running Docker Containers as Current Host User

Sun, 05 Aug 2018 00:00:00 -0500

ed: If you want to jump right to the solution, jump ahead to Ok so what actually works?.

Docker is an excellent tool for local web development. It allows creating non-trivial environments without polluting the local system with tools.

There are still some things that make working with it just a tad bit harder than necessary.

Today’s topic involves running Docker containers using the local host system’s current logged-in user.

The Problem

You are working on a project that requires Node NPM, PHP Composer or a similar tool that downloads or compiles outside dependencies or assets for you.

You do not want to install the language (PHP, Node) locally to run this tool, so you choose to run a Docker container. Here is how it would look like to spin up a container with PHP Composer support:

docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ jtreminio/php:7.2 composer require psr/log 

This starts a temporary container using my PHP 7.2 image that has Composer pre-installed and runs composer require psr/log.

It generates the following:

$ ls -la total 20K drwxrwxr-x. 3 jtreminio jtreminio 4.0K Aug 4 19:34 ./ drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug 4 19:31 ../ drwxr-xr-x. 4 root root 4.0K Aug 4 19:34 vendor/ -rw-r--r--. 1 root root 53 Aug 4 19:34 composer.json -rw-r--r--. 1 root root 2.1K Aug 4 19:34 composer.lock 

Although it works as intended, the problem is that the directories and files generated by Composer are root-owned.

Since I do not use root, attempting to do anything other than read the files is blocked:

$ echo '' > composer.json bash: composer.json: Permission denied 

You have to sudo to edit or remove everything. This quickly gets old when your project uses Composer, Webpack, Yarn, etc because your system becomes littered with root-owned files and directories.

The Reason

Docker on Linux runs as a daemon. The official installation instructions recommend installing as root and selectively adding users to the docker group so they can run all Docker commands.

$ ps -fe | grep dockerd 255:root 2356 1 0 Aug03 ? 00:04:06 /usr/bin/dockerd 

When you create a new container it does not get created as your current user, but as root, which the daemon is running under.

We can verify that the container runs as root with user/group ID 0:0:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ jtreminio/php:7.2 whoami root 

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ jtreminio/php:7.2 \ bash -c "echo \$(id -u \${USER}):\$(id -g \${USER})" 0:0 

So run as your local user, right?

When you run Docker containers you can specify a user ID, plus a group ID. It is easy enough to do:

docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 composer require psr/log 

This generates the following:

$ ls -la total 20K drwxrwxr-x. 3 jtreminio jtreminio 4.0K Aug 4 20:09 ./ drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug 4 19:31 ../ drwxr-xr-x. 4 jtreminio jtreminio 4.0K Aug 4 20:09 vendor/ -rw-r--r--. 1 jtreminio jtreminio 53 Aug 4 20:09 composer.json -rw-r--r--. 1 jtreminio jtreminio 2.1K Aug 4 20:09 composer.lock 

In my system, my user jtreminio has user ID 1000 and group ID 1000, so the new line

-u $(id -u ${USER}):$(id -g ${USER})

gets interpreted as

-u 1000:1000

This does exactly what we want, but of course there is a catch: the container user is no longer root, or whatever the author decided to use. On Composer and NPM this simply means any internal cache directories cannot be written to since they are root-owned, but that really is not much of a problem because we are tearing the containers down as soon as they finish running what we told them to.

The Composer container above is deleted as soon as composer require psr/log finishes executing.

What if you want to run a web app with PHP-FPM? It must be able to create its PID file at /var/run/php-fpm.pid, if you are using file-based sessions it must write to /var/lib/php/sessions. Any number of things that require root or a predefined user will no longer work because the container is running using your user/group ID:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 whoami whoami: cannot find name for user ID 1000 

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 \ bash -c "echo \$(id -u \${USER}):\$(id -g \${USER})" 1000:1000 

If you try to do anything that requires elevated permissions or a specific user, you will be denied:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 ls -la /var/lib/php/sessions total 8 drwxr-xr-x. 1 www-data www-data 4096 Jul 9 12:35 . drwxr-xr-x. 1 root root 4096 Jul 26 09:08 .. 

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 touch /var/lib/php/sessions/foo touch: cannot touch '/var/lib/php/sessions/foo': Permission denied 

The above means running the PHP-FPM daemon as your local user will quickly encounter permissions issues. In this case it is because /var/lib/php/sessions is owned by www-data:www-data which most likely does not share your local user’s IDs:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ jtreminio/php:7.2 \ bash -c "echo \$(id -u www-data):\$(id -g www-data)" 33:33 

OK, run it as an non-root internal user?

So far we have found that

root works great inside the container but is annoying to work with on the host system, and
your local user works great on your host system, but will quickly run into permission problems inside the container.

What if we try to run the containers as a non-root user that has required permissions to write inside the container directories?

Try with the container’s internal www-data user:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u www-data \ jtreminio/php:7.2 \ bash -c "touch /var/lib/php/sessions/foo && echo \$?" 0 

As expected this worked fine, but if you try running Composer:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u www-data \ jtreminio/php:7.2 composer require psr/log [ErrorException] file_put_contents(./composer.json): failed to open stream: Permission denied 

The problem is that the internal container user www-data with user/group ID 33:33, does not have write permissions to my host’s current directory:

$ ls -la total 8.0K drwxrwxr-x. 2 jtreminio jtreminio 4.0K Aug 4 20:34 ./ drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug 4 19:31 ../ 

We might as well add another grievance to our list:

non-root internal user with required permissions works great inside the container but completely falters on host system with volumes.

Why is this happening?

Host and containers do not share users and groups.

On my local system, there is no www-data user:

$ groups www-data groups: ‘www-data’: no such user 

Similarly, if I try using my direct username to run the container I see errors:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u jtreminio \ jtreminio/php:7.2 whoami docker: Error response from daemon: linux spec user: unable to find user jtreminio: no matching entries in passwd file. 

As far as the container is aware, it is its own separate system with its own list of users at /etc/passwd and list of groups at /etc/group. It has no idea about any users or groups that exist on the host system. Likewise, the host does not know about users in the container. By convention root is 0:0 which will match any Linux system, but anything else is up to each distro and each image author.

Try sharing `/etc/passwd`!

Now we know that the

host system does not know about container’s /etc/passwd, and
container does not know about the host system’s /etc/passwd

What happens if we bind the host’s /etc/passwd to the container’s?

Not much as you would think, actually. My host system still does not have a user matching 33:33, so even if I bind the /etc/passwd into the container, attempting to write to www-data-owned directories still will not work.

This is because Linux permissions are not name-based, but ID-based.

For example, on my local system:

$ ls -lan total 8.0K drwxrwxr-x. 2 1000 1000 4.0K Aug 4 20:34 ./ drwxr-xr-x. 7 1000 1000 4.0K Aug 4 19:31 ../ 

and in the container:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ -u $(id -u ${USER}):$(id -g ${USER}) \ jtreminio/php:7.2 ls -lan /var/lib/php/sessions total 8 drwxr-xr-x. 1 33 33 4096 Jul 9 12:35 . drwxr-xr-x. 1 0 0 4096 Jul 26 09:08 .. 

In the container www-data:www-data is simply an alias to 33:33.

User namespaces

The container does not care what user and group name is used, it simply wants the IDs to match.

There is a concept in Docker Engine called User Namespaces. Here is a great introduction to them.

The concept boils down to mapping the internal container user/group IDs to reflect different values.

For example, you can tell Docker to use your current user/group ID as the “floor” for container IDs. In my example, my jtreminio account with 1000:1000 would map directly to 0:0 in a container.

In other words, we tell Docker to consider our current user on the host as root in containers! Local system user ID 1000 maps directly to container user ID 0.

User namespaces would allow us to run all containers as root internally which would completely eliminate any permission issues, and any generated files and directories on shared volumes would be owned by the host user/group so we would no longer need to sudo to edit or delete them.

This sounds exactly what we need, right? There is a catch (because of course there is)… in fact, multiple catches!

You are running the container as root

Normally this would be a security concern. Countless articles and guides strongly recommend not running as root in containers.

In this case, this is not the problem, because the container’s internal root maps to our current, non-root user on host.

The problem actually comes down to root being able to do anything and everything in the container, which opens the door to unforeseen permissions issues when you deploy to production, where you would not be using user namespaces.

In production, most containers will run as non-root, with permissions specifically tailored for that user/group ID. Attempting to do anything the user is not permitted will rightfully result in permissions denied errors. This is a safety feature you want, and running as root completely ignores this.

You might run as root on your local system and everything works perfectly fine. You are happy, files are generated as expected, all actions are permitted and you are a happy developer. Once you deploy to production you may suddenly realize that you were developing on a system very much unlike production. Something that worked flawlessly on your development system can easily break on production due to insufficient permissions for whatever user is running the container service.

Your user/group ID is the floor, it goes up!

Your user/group ID maps directly to 0:0 in the container, but there are more accounts than just the one root and none of them are mapped to your host user.

This means that while 0:0 maps to my host system’s 1000:1000, the container’s www-data user with IDs 33:33 would map to my host system’s 1033:1033, which does not exist.

Anything done as non-root in the container will run against the same issues we saw earlier: what might be considered sufficient permissions inside the container will probably not work the same on your host.

My host’s directories are still owned by 1000:1000 and a user with 1033:1033 will be denied.

All containers on your system are affected

If the previous two reasons are not enough, the fact that Docker does not come with User Namespaces support enabled by default, to enable it requires editing a JSON config file and restarting the Docker daemon, and now all your existing and future containers will implement namespaces whether you want them to or not should be enough to make you question if it is worth the hassle.

Ok so what actually works?

The only sure-fire way of resolving all previous issues and getting on with Making the World a Better Place is completely replacing the internal user/group IDs with known, good values.

In the container, www-data maps to 33:33. Why not change it so it maps to my host system’s jtreminio with 1000:1000?

In other words, make the container’s www-data user/group ID be 1000:1000.

You cannot change an existing account’s user/group IDs. Remember, the user and group names are simply aliases to the IDs. You can rename www-data to anything you want but the IDs will not change.

The only thing you can do is delete and recreate the user/group completely.

For this we will need to create a Dockerfile. We need to recreate the user before any potential entrypoint scripts are run. Unknown issues can and will occur if you attempt to delete a user/group while it is in use by another process.

First, delete the user and group:

FROM jtreminio/php:7.2 RUN userdel -f www-data &&\  if getent group www-data ; then groupdel www-data; fi 

We test if the group actually exists before trying to delete, to avoid possible errors when the group does not exist.

Next, we can generate the user and group with our IDs:

FROM jtreminio/php:7.2 ARG USER_ID=1000 ARG GROUP_ID=1000 RUN userdel -f www-data &&\  if getent group www-data ; then groupdel www-data; fi &&\  groupadd -g ${GROUP_ID} www-data &&\  useradd -l -u ${USER_ID} -g www-data www-data &&\  install -d -m 0755 -o www-data -g www-data /home/www-data 

After we delete the user and group, we recreate it with the defined values.

1000 is my user and group ID, and now the container’s www-data has the same.

Remember to use the -l flag with useradd! There is an amazingly fun issue where a high UID value will generate huge log files and freeze your system! Click here for all the juicy details.

We also take the time to generate a home directory for our new user. This makes it much easier to have your container perform SSH actions using your host’s SSH keys as long as you bind a volume.

Even though we are done with recreating the user and group, there is a problem: Any files and directories that were owned by the www-data user will not automatically point to the new ID values. Since the names are just labels, /var/lib/php/sessions is still owned by 33:33, not matching our spiffy new 1000:1000!

This is the part where you need to know just enough about the container you want to run that you can pinpoint which files and directories you need to update the owner for.

Thankfully, the rallying cry of containers is to have them do one single thing, which limits the number of places we need to update. In this image, it is not very many at all:

FROM jtreminio/php:7.2 ARG USER_ID=1000 ARG GROUP_ID=1000 RUN userdel -f www-data &&\  if getent group www-data ; then groupdel www-data; fi &&\  groupadd -g ${GROUP_ID} www-data &&\  useradd -l -u ${USER_ID} -g www-data www-data &&\  install -d -m 0755 -o www-data -g www-data /home/www-data &&\  chown --changes --silent --no-dereference --recursive \  --from=33:33 ${USER_ID}:${GROUP_ID} \  /home/www-data \  /.composer \  /var/run/php-fpm \  /var/lib/php/sessions 

You can figure out what you need by looking at the Dockerfile used to generate the image. For the jtreminio/php:7.2 image we can see that /.composer, /var/run/php-fpm and /var/lib/php/sessions directories need updated.

Finally, it does not hurt to be explicit about the user we want to run the container:

FROM jtreminio/php:7.2 ARG USER_ID=1000 ARG GROUP_ID=1000 RUN userdel -f www-data &&\  if getent group www-data ; then groupdel www-data; fi &&\  groupadd -g ${GROUP_ID} www-data &&\  useradd -l -u ${USER_ID} -g www-data www-data &&\  install -d -m 0755 -o www-data -g www-data /home/www-data &&\  chown --changes --silent --no-dereference --recursive \  --from=33:33 ${USER_ID}:${GROUP_ID} \  /home/www-data \  /.composer \  /var/run/php-fpm \  /var/lib/php/sessions USER www-data 

If you run the container now, you will automatically become the www-data user which now has user/group ID 1000:1000. Any files generated by this container will be owned by my host system’s user.

What should we do about the hard-coded IDs, though? Your coworkers may have different IDs on their machines.

Make it dynamic

You can pass arguments when building an image using --build-arg.

Update the Dockerfile first:

FROM jtreminio/php:7.2 ARG USER_ID ARG GROUP_ID RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then \  userdel -f www-data &&\  if getent group www-data ; then groupdel www-data; fi &&\  groupadd -g ${GROUP_ID} www-data &&\  useradd -l -u ${USER_ID} -g www-data www-data &&\  install -d -m 0755 -o www-data -g www-data /home/www-data &&\  chown --changes --silent --no-dereference --recursive \  --from=33:33 ${USER_ID}:${GROUP_ID} \  /home/www-data \  /.composer \  /var/run/php-fpm \  /var/lib/php/sessions \ ;fi USER www-data 

We make both USER_ID and GROUP_ID optional. If both are not defined, we skip the whole user delete/recreate process completely. This Dockerfile can now be used for both dev and production purposes.

To build the above image you would do:

$ docker image build \ --build-arg USER_ID=$(id -u ${USER}) \ --build-arg GROUP_ID=$(id -g ${USER}) \ -t php_test \ . Sending build context to Docker daemon 68.1kB Step 1/5 : FROM jtreminio/php:7.2 ---> 88efaa8cad72 Step 2/5 : ARG USER_ID ---> Running in 51d424a6d618 Removing intermediate container 51d424a6d618 ---> 3feace11551d Step 3/5 : ARG GROUP_ID ---> Running in aaaceff5ee4e Removing intermediate container aaaceff5ee4e ---> 9f110b0f92c2 Step 4/5 : RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then userdel -f www-data && if getent group www-data ; then groupdel www-data; fi && groupadd -g ${GROUP_ID} www-data && useradd -l -u ${USER_ID} -g www-data www-data && install -d -m 0755 -o www-data -g www-data /home/www-data && chown --changes --silent --no-dereference --recursive --from=33:33 ${USER_ID}:${GROUP_ID} /home/www-data /.composer /var/run/php-fpm /var/lib/php/sessions ;fi ---> Running in b516a3ab4600 changed ownership of '/.composer/keys.tags.pub' from 33:33 to 1000:1000 changed ownership of '/.composer/keys.dev.pub' from 33:33 to 1000:1000 changed ownership of '/.composer' from 33:33 to 1000:1000 changed ownership of '/var/run/php-fpm' from 33:33 to 1000:1000 changed ownership of '/var/lib/php/sessions' from 33:33 to 1000:1000 Removing intermediate container b516a3ab4600 ---> c126c9f07252 Step 5/5 : USER www-data ---> Running in 76831280f239 Removing intermediate container 76831280f239 ---> 5c237c8e46cd Successfully built 5c237c8e46cd Successfully tagged php_test:latest 

Try running Composer with the new image:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ php_test:latest composer require psr/log 

and check out the result:

$ ls -lan total 24K drwxrwxr-x. 3 1000 1000 4.0K Aug 4 22:50 ./ drwxr-xr-x. 7 1000 1000 4.0K Aug 4 19:31 ../ drwxr-xr-x. 4 1000 1000 4.0K Aug 4 22:50 vendor/ -rw-rw-r--. 1 1000 1000 545 Aug 4 22:48 Dockerfile -rw-r--r--. 1 1000 1000 53 Aug 4 22:50 composer.json -rw-r--r--. 1 1000 1000 2.1K Aug 4 22:50 composer.lock 

Try creating a file in a protected directory:

$ docker container run --rm \ -v ${PWD}:/var/www \ -w /var/www \ php_test:latest \ bash -c "touch /var/lib/php/sessions/foo && echo \$?" 0 

User Docker Compose

You can take the docker image build command and save it to a bash file for easy execution.

However, most folks would rather use Docker Compose for local development.

Here is what your docker-compose.yml might look like:

version: '3.2' services: php: build: context: . dockerfile: Dockerfile args: USER_ID: $(id -u ${USER}) GROUP_ID: $(id -g ${USER}) volumes: - ${HOME}/.composer:/.composer - ${PWD}:/var/www ports: - 9000:9000 

Unfortunately, Docker Compose cannot parse commands, so $(id -u ${USER}) would not work in the YAML file. Thankfully we can use .env file to pass values:

# docker-compose.yml version: '3.2' services: php: build: context: . dockerfile: Dockerfile args: USER_ID: ${USER_ID:-0} GROUP_ID: ${GROUP_ID:-0} volumes: - ${HOME}/.composer:/.composer - ${PWD}:/var/www ports: - 9000:9000 

; .env USER_ID=1000 GROUP_ID=1000 

With a single docker-compose up -d --build we are up and running:

$ docker-compose up -d --build Creating network "temp_default" with the default driver Building php Step 1/5 : FROM jtreminio/php:7.2 ---> 88efaa8cad72 Step 2/5 : ARG USER_ID ---> Running in 1b406c5797aa Removing intermediate container 1b406c5797aa ---> c61d4e7b14b9 Step 3/5 : ARG GROUP_ID ---> Running in 14ac56de9dc8 Removing intermediate container 14ac56de9dc8 ---> 576ada9d6dbe Step 4/5 : RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then userdel -f www-data && if getent group www-data ; then groupdel www-data; fi && groupadd -g ${GROUP_ID} www-data && useradd -l -u ${USER_ID} -g www-data www-data && install -d -m 0755 -o www-data -g www-data /home/www-data && chown --changes --silent --no-dereference --recursive --from=33:33 ${USER_ID}:${GROUP_ID} /home/www-data /.composer /var/run/php-fpm /var/lib/php/sessions ;fi ---> Running in 632e67cdc854 changed ownership of '/.composer/keys.tags.pub' from 33:33 to 1000:1000 changed ownership of '/.composer/keys.dev.pub' from 33:33 to 1000:1000 changed ownership of '/.composer' from 33:33 to 1000:1000 changed ownership of '/var/run/php-fpm' from 33:33 to 1000:1000 changed ownership of '/var/lib/php/sessions' from 33:33 to 1000:1000 Removing intermediate container 632e67cdc854 ---> 1bb2e0bbff3c Step 5/5 : USER www-data ---> Running in 8efe4779c454 Removing intermediate container 8efe4779c454 ---> b3bd40096b94 Successfully built b3bd40096b94 Successfully tagged temp_php:latest Creating temp_php_1 ... done 

We now have a PHP-FPM container running with my system user/group ID 1000:1000 waiting for connections from Nginx or Apache.

What containers need this?

Not all containers actually require these steps. For simpler images like the official Nginx or Apache that do not generate files you would be interested in editing while in development, you can skip this and let the container run as the normal internal user.

For containers that only fetch dependencies to local disk, like PHP Composer or Node NPM, you can get by with simply passing your user/group ID directly. The container will complain about a user not existing for those values, but will otherwise work fine. Cache directories may need a little bit of work.

Containers with long-running daemons that generate a large amount of files, like parsing SCSS to CSS, will benefit you greatly to implement this concept.

MacOS and Windows users…

You do not actually need to do any of this. Like, this whole article, you can skip it. Docker does not actually run natively on either of these operating systems, but transparently in a virtual machine.

This virtual machine is run under whatever user installed Docker, which is usually not root on MacOS or administrator on Windows.

While you do not need this, it is still a good idea to implement it. Your coworkers that run Linux as their main OS will benefit greatly, and if you ever make the switch you will already have your Docker stuff up to date.

You will also only require a single Dockerfile between all your coworkers.

Wrapping it up

While you can run all containers as the default internal user, dedicating just a few minutes per image to switch to your user/group ID will help eliminate one fairly big point of frustration.

Overall Docker is an amazing technology but little things like this can mar an otherwise enjoyable experience. Knowing how to smooth the bumps will help make Docker’s promise of Nirvana come true.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

Traefik on Docker for Web Developers

Tue, 31 Jul 2018 00:00:00 -0500

Over the last 5+ years I have done all my development on local virtual machines managed by Vagrant and provisioned by Puppet. I even created a fairly well-received FOSS called PuPHPet.

At the end of 2017 I started really looking into containers, and as of January started working on what will become PuPHPet’s successor, Dashtainer.

While this is not a post on containers in general, or Dashtainer, the thing I will talk about fits in perfectly in the container world and is used heavily in Dashtainer.

Problems with Docker

While Docker is an amazingly useful tool, it does not come without its own set of problems.

One of the biggest is what I call Docker port dancing. In Docker, you can bind a port on your host to forward to a container.

For example, you can bind port 80 on host to port 80 on a container, so going to http://localhost will automatically forward the request to the container. In this way you can bind a webserver (80), PHP-FPM (9000) and MySQL (3306) and very quickly have a complete working environment on your local machine without having to actually install any of those tools, existing only within their containers.

If you ever have only a single project, this may be fine, but once you start spinning up more projects you quickly realize the biggest limitation: you cannot bind a port on host multiple times.

If port 80 is mapped to web-server-A you must choose another port to bind for web-server-B and web-server-C. This can quickly get old because you must remember that http://localhost goes to A, http://localhost:81 goes toB and http://localhost:82 goes to C. Of course the actual port you bind is completely up to you so you can do 8080 or 8000 or any unused port on your local machine.

On virtual machines this problem does not really occur because you can assign a static IP address to your servers, and bind it to your system’s hosts file (/etc/hosts). Containers are ephemeral by nature and do not normally get created on your host’s network but rather private networks with their own random IP addresses within special ranges. However, you must edit /etc/hosts for every VM you spin up and the list grows with the number of projects you handle.

Træfik solves both of these problems, first by removing the need to use ports in URLs and second by not needing you to edit /etc/hosts at all.

What is Træfik?

Træfik (pronounced traffic, spelled Traefik from now on) is a reverse proxy / load balancer similar to HAProxy or Nginx in reverse proxy mode.

Simply put, as a reverse proxy it monitors traffic to specified ports (80,443) and routes traffic to the proper endpoint.

Traefik includes baked-in support for Docker and you can configure it almost fully through flags, with no need for config files. It supports hot-loading and automatically detects changes to environment. Best of all, it supports Let’s Encrypt right out of the box.

Traefik runs as a separate container and this single container can work across any number of separate projects you want. It works by listening to the Docker daemon and reacting to labels you define for each container.

Creating a Traefik Container

First create the network Traefik will use:

1 docker network create --driver bridge traefik_webgateway

Then create the actual Traefik container:

docker container run -d --name traefik_proxy \ --network traefik_webgateway \ -p 80:80 \ -p 8080:8080 \ --restart always \ --volume /var/run/docker.sock:/var/run/docker.sock \ --volume /dev/null:/traefik.toml \ traefik --api --docker 

With the above we,

tell Traefik to watch to a specific network (--network traefik_webgateway) for new containers. Any containers attached to this network can be monitored by Traefik.
tell Traefik to attach itself to the Docker daemon (--volume /var/run/docker.sock:/var/run/docker.sock). This is what allows Traefik to listen to the above network and read other containers’ labels.
decline the use of a config file (--volume /dev/null:/traefik.toml) to highlight Traefik’s ability to be completely configured via arguments.
bind host’s ports 80 and 8080 (-p 80:80 and -p 8080:8080) to Traefik.
enable the Traefik GUI dashboard (--api)

How Traefik Solves the Port Dance

Traefik can automatically pick up any containers that use the traefik_webgateway network and reads labels applied.

For example, to spin up a new Nginx container you could do something like:

docker run -d --name some-nginx \ -v ${PWD}:/usr/share/nginx/html:ro \ --network traefik_webgateway \ --label traefik.docker.network=traefik_webgateway \ --label traefik.frontend.rule=Host:some-nginx.localhost \ --label traefik.port=80 \ nginx:alpine 

Here we tell Traefik that this container’s hostname is some-nginx.localhost and it receives traffic on port 80.

If you open some-nginx.localhost in Chrome¹ you should see the Nginx container responding.

Using hostnames directly without having to append port numbers to them makes working with Docker containers much easier than having to remember which port goes with which project and which container.

Traefik GUI

Above I mentioned the Traefik GUI dashboard. It listens on port 8080 so simply open localhost:8080 and you will see all the containers Traefik is currently monitoring for changes.

Non-port 80 Example

Some containers do not listen on port 80 by default. This is fine because you can forward any traffic on port 80 on the container’s hostname to the specified port.

For example, MailHog’s GUI sits on port 8025. You can run it with:

docker run -d --name some-mailhog \ --network traefik_webgateway \ --label traefik.docker.network=traefik_webgateway \ --label traefik.frontend.rule=Host:mailhog.localhost \ --label traefik.port=8025 \ mailhog/mailhog 

Even though the container listens to 8025, opening mailhog.localhost will automatically forward traffic to the proper port.

If you open the Traefik dashboard you will see this new container listed.

When you run docker container rm -f some-mailhog it will automatically be removed.

Built-in Let’s Encrypt Support

Once you are ready to go live with your website configuring Traefik to automatically request and maintain a valid Let’s Encrypt SSL certificate is fairly easy!

There are some extra arguments you must define but nothing too foreign:

docker container run -d --name traefik_proxy \ --network traefik_webgateway \ -p 80:80 \ -p 443:443 \ -p 8080:8080 \ --restart always \ --volume /var/run/docker.sock:/var/run/docker.sock \ --volume /dev/null:/traefik.toml \ --volume /root/acme.json:/root/acme.json \ traefik --docker --logLevel=INFO \ --acme \ --acme.acmelogging \ --acme.dnschallenge=false \ --acme.entrypoint="https" \ --acme.httpchallenge \ --acme.httpChallenge.entryPoint="http" \ --acme.onhostrule=true \ --acme.storage="/root/acme.json" \ --acme.email="your-email-here@example.com" \ --entrypoints="Name:http Address::80 Redirect.EntryPoint:https" \ --entrypoints="Name:https Address::443 TLS" \ --defaultentrypoints="http,https" 

Since we will handle SSL traffic we add -p 443:443 to the ports list, and since this is for a live server we remove the dashboard (--api).

Let’s Encrypt stores its data in files and it requires special permissions before it will work. Unfortunately this is the only thing you cannot do via only CLI arguments. You must create this file and set permissions properly before running the above command.

Simply do touch /root/acme.json && chmod 600 /root/acme.json and you are all set.

We redirect all 80 traffic to 443 to encrypt all traffic (as you should already be doing!). Let’s Encrypt cert is generated after it pings http://your-website.com/.well-known/acme-challenge to verify ownership.

Traefik will take care of keeping all certs up to date.

Explicitly Disable Traefik for Non-HTTP Services

It seems by default Traefik will attempt to generate a Let’s Encrypt cert for all containers, even if the containers are not on the Traefik network.

To prevent Let’s Encrypt errors from breaking your build, you must explicitly disable Traefik on containers that do not need a cert, like a PHP-FPM container.

It is easy enough:

docker run -d --name php-fpm \ --network private \ --label traefik.enable=false \ your-php/image 

The above is not required on dev since you will not be generating SSL certs.

Limitations

Traefik only handles HTTP/HTTPS traffic. It cannot currently handle TCP/UDP.

This means any database containers cannot be aliased to a hostname, and you will need to do the port dance on these services.

This is an unfortunate limitation, and the GitHub issue has been open for a while without much movement.

This does not mean your services cannot communicate with each other via normal Docker hostnames. For example, a service named php-fpm on network foobar can still communicate with a service named mysql on the same foobar network, using the hostname mysql.

You, however, cannot access mysql from your host via a GUI like Sequel Pro or MySQL Administrator. You can either bind a port from host to container, or use something like Adminer.

Via docker-compose.yml

If you prefer to try this out using Docker-Compose, create the following 3 files:

; traefik/.env COMPOSE_PROJECT_NAME=traefik 

# traefik/docker-compose.yml version: '3.2' services: proxy: image: traefik command: --api --docker --docker.domain=docker.localhost --logLevel=DEBUG networks: - webgateway ports: - "80:80" - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock - /dev/null:/traefik.toml networks: webgateway: driver: bridge 

# app/docker-compose.yml version: '3.2' networks: private: public: external: name: traefik_webgateway services: nginx: image: nginx:alpine labels: - traefik.backend=nginx - traefik.docker.network=traefik_webgateway - traefik.frontend.rule=Host:some-nginx.localhost - traefik.port=80 networks: - private - public mailhog: image: mailhog/mailhog labels: - traefik.backend=mailhog - traefik.docker.network=traefik_webgateway - traefik.frontend.rule=Host:mailhog.localhost - traefik.port=8025 networks: - private - public 

Run docker-compose up -d in the traefik directory, then the app directory.

Shameless Plug

If you are new to the world of containers, I have created a FOSS called Dashtainer to help you quickly generate and run containers tailored to your app’s requirements.

If you give it a go, I would love to hear from you!

Wrapping it up

Docker has brought containers to the mainstream, but little gotchas like port dancing can be frustrating to new users. Hopefully with this small tutorial you are able to get up and running and get back to developing your Make the World a Better Place app.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

I specify Chrome because as of this writing I believe it is the only browser that will always resolve any *.localhost to the loopback interface. This means you do not need to touch the /etc/hosts file, the hostname will work automatically. If you do not want to use Chrome then you must install dnsmasq on MacOS or Acrylic on Windows. ↩

Developing at Full Speed with Xdebug

Mon, 23 Jul 2018 00:00:00 -0500

A quick history

ed: If you want to jump right to the solution, jump ahead to Nginx map.

Docker for Mac is very slow.

It is so slow that I purchased a new Dell XPS laptop and for the first time in 6 years am now using a non-MacOS (Fedora) machine as my daily driver.

Not everyone has the luxury of switching their OS, though, and they are stuck on slow Docker.

A normal Symfony 2.4 application will commonly see between 400ms-750ms response times in development mode, without Xdebug installed. If Xdebug is activated response times of 1,200ms+ can frustrate even the most devoted Xdebug fan.

Before switching to Fedora I tried everything I could to minimize Xdebug’s impact on performance. I have been using Xdebug for several years and while I always felt the benefits of Xdebug far out-weighed the extra slowness, I knew there had to be a better way.

How Xdebug decides to run

There are several ways of enabling Xdebug for a specific session. The more popular ways are using cookies, like those generated by PhpStorms bookmarklets. You can also kick Xdebug off via CLI to debug command line scripts without a web portion.

All methods share the same requirement: Xdebug must be installed and loaded on the system to work.

The all also let the PHP layer of your application decide whether to enable Xdebug for the current session.

If you take a look at PhpStorm’s bookmarklets, the code is actually quite simple:

javascript:(/** @version 0.5.2 */function() {document.cookie='XDEBUG_SESSION='+'xdebug'+';path=/;';})() 

It simply sets a cookie named XDEBUG_SESSION and sets a value to it. By default PhpStorm wants to use… phpstorm, but I always set it to xdebug as above.

Knowing this,would it be possible to move the decision one layer above, out of PHP’s hands? Instead of PHP reading for the XDEBUG_SESSION cookie and acting on it, do it somewhere else.

Docker configuration

A normal Docker PHP application looks like:

1 webserver like Nginx
1 PHP container with Xdebug installed

The problem so far has been that the PHP side of things is slowing everything down, due to Xdebug being installed.

What if instead of the above, we had:

1 webserver like Nginx
1 PHP container with Xdebug installed (named php_xdebug)
1 PHP container without Xdebug installed (named php)

The trick here is making the decision to invoke Xdebug before PHP becomes aware that a new request is being processed. Thankfully, Nginx’s map can help us tremendously!

Nginx map

Nginx maps are quite simple, and what I came up with to handle the Xdebug cookie requirement is this:

map $cookie_XDEBUG_SESSION $my_fastcgi_pass { default php; xdebug php_xdebug; } 

In PHP the above would look like this:

<?php switch ($cookie_XDEBUG_SESSION) { case 'xdebug': $my_fastcgi_pass = 'php_xdebug'; break; default: $my_fastcgi_pass = 'php'; } 

$cookie_XDEBUG_SESSION is the cookie set by PhpStorm’s bookmarklet, and $my_fastcgi_pass is the server Nginx will use for this request.

A full Symfony 3 Nginx config would look like this:

# default Docker DNS server resolver 127.0.0.11; map $cookie_XDEBUG_SESSION $my_fastcgi_pass { default php; xdebug php_xdebug; } server { listen *:8080 default_server; server_name _; root /var/www/public; autoindex off; location / { try_files $uri /app.php$is_args$args; } location ~ ^/(app_dev|config)\.php(/|$) { set $path_info $fastcgi_path_info; fastcgi_pass $my_fastcgi_pass:9000; fastcgi_split_path_info ^(.+\.php)(/.*)$; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param DOCUMENT_ROOT $realpath_root; } location ~ ^/app\.php(/|$) { set $path_info $fastcgi_path_info; fastcgi_pass php:9000; fastcgi_split_path_info ^(.+\.php)(/.*)$; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param DOCUMENT_ROOT $realpath_root; internal; } location ~ \.php$ { return 404; } } 

You can see how we’re actually using the result of the map at fastcgi_pass $my_fastcgi_pass:9000;.

The Nginx part is simple, but critical. Nginx reads the cookie and sends traffic either to the php server or the php_xdebug server. There is nothing magical about this, and PHP is not aware that a request is being made until Nginx has made the decision to route the request to its server. The other server would remain blissfully unaware of the request.

Good solution or hacky workaround?

Your normal workflow with Xdebug would look like this:

You enable the Xdebug cookie in your browser
You create a breakpoint and enable the listener in your IDE
You load the page in your browser, Xdebug and your IDE connect and stop at the breakpoint defined.

With my Nginx map solution, your workflow remains exactly the same. It does not change!

But, instead of Xdebug slowing down every single request, whether you have set a breakpoint or not, whenever you do not have the XDEBUG_SESSION cookie enabled, you are hitting the non-Xdebug server! In real-world testing this means that the majority of my requests are now routed directly to the PHP server that does not have Xdebug installed.

On my MacBook Pro requests see a normal response time of sub-500ms. This is still not great (because, Docker), but is worlds better than every single request being 1,000ms+!

When I do want to enable debugging, I set my breakpoint, enable the XDEBUG_SESSION cookie by using PhpStorm’s bookmarklets, and Nginx routes my request to the PHP server with Xdebug installed.

When this happens, I have a breakpoint enabled and will be looking at my IDE, so response times do not matter at all. The biggest (and only) painpoint that Xdebug introduces is completely eliminated.

What took so long?

Until 6 months ago I was using Vagrant virtual machines for all my development. My FOSS PuPHPet is evidence that I was all-in on VMs.

However, since completely making the switch to Docker containers, my workflow has changed for the better, even with the slowness Docker for Mac shows.

Docker containers have the benefit of being incredibly light-weight compared to virtual machines, and this makes spinning up two PHP containers for each of my projects completely feasible.

This solution would have been much heavier on virtual machines. On containers it is easier, simpler and faster.

Downsides

I have not found any downsides to this technique. If you use file-based sessions (why?) then you’ll have to share the volumes between the PHP containers.

Test it out

If you want to try this out in a real-world project, and are not familiar enough with Docker to do so quickly, then you can clone my new FOSS Dashtainer repo and run the bin/init script to get you up and running in a single step.

On a very related note, if you want to get started with Docker but need some help for the initial hurdles, my FOSS Dashtainer is the successor to my other FOSS PuPHPet and is aimed at developers who need an introduction to the strange new world of containers.

edit: I have created a barebones repo for you to test this out. Clone my jtreminio/blog repo and run init inside the developing-at-full-speed-with-xdebug directory.

Bonus: Identical configurations in Windows, MacOS and Linux

Docker has a special hostname, host.docker.internal that points to the host machine. Unfortunately this only works on Windows and MacOS, Docker on Linux does not have this feature.

This makes setting the xdebug.remote_host value in the PHP INI annoying becuase not all of your team can simply spin up your Docker container and have Xdebug connect back to the host automatically if they are not on Linux.

There is xdebug.remote_connect_back which does just that, though, except it has always been a double-edged sword because it would attempt to connect back for every single request. My solution above resolves this completely!

Simply set xdebug.remote_connect_back=1 in your INI and Xdebug will connect back to your IDE everytime you enabled the session using PhpStorm’s bookmarklets.

Wrapping it up

With this small technique you gain the ability to develop at full speed without sacrificing modern tools like the incredible Xdebug. Usually with these sorts of things there’s also some downside, but all I see is positives.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!

Juan Treminio - Senior Web Developer Blog

OpenAPI Tutorial Part III: Paths and Basic Request Data

Is Redoc Documentation Server Running?

Adding Paths

Definition of a Path

Your First Path Definition

Break It Down - Path Annotations

Verb + Path

Summary & Description

Operation ID

Tags

Adding a Body Definition

Body Annotations

Break It Down - Body Annotations

Referencing a Schema

Break It Down - RequestBody Annotations

At a Glance - ref

Enum Property Types

But Wait, There’s More

Wrapping It Up

OpenAPI Tutorial Part II: Common API Example

jtreminio/openapi-tutorial

Controllers

Pet Controller

Customer Controller

Order Controller

Models

Pet Model

Customer Model

Order Model

Why This API?

Verbs

Enums

Nested Objects

Arrays

Polymorphism

Wrapping It Up

OpenAPI Tutorial Part I: Introduction to OpenAPI

OpenAPI? I Thought It Was Swagger?

What Can I Do With OpenAPI?

Documentation

SDK Generation

Before Getting Started, Tools to Use

Generating Your Definitions File

Generating Documentation

SDK Generation

Choosing an API to Implement

Initial Setup

Organizing Our Code

Setting Up zircote/swagger-php

Setting Up redoc-cli

Create OpenApi Entry File

Initial Generation

Final Project Structure

Extra Credit

Wrapping It Up

All-in-One PHP-FPM + Nginx/Apache Containers

Managing Multiple Services Under Docker

Setting up Nginx

Setting up PHP-FPM

Bonus: Setting up Apache

Try It Out

Wrapping It Up

PHP Modules Toggled via Environment Variables

Why Not Use Environment Variables To Load Modules?

Meet PHP_INI_SCAN_DIR

Loading Modules with PHP_INI_SCAN_DIR

Bonus: Separate FPM and CLI PHP_INI_SCAN_DIR Values

About the Image Size…

Want to Try It Out?

Wrapping It Up

Docker PHP/PHP-FPM Configuration via Environment Variables

Our Goals

Quick Introduction to Docker Images

PHP INI with Environment Variables (env vars)

Caveats

Building Docker Image with Env Vars

Create a Parent Image

Child Images

Removing CMD from Dockerfile

At a Glance - `ref`

Setting Up `zircote/swagger-php`

Setting Up `redoc-cli`

Meet `PHP_INI_SCAN_DIR`

Loading Modules with `PHP_INI_SCAN_DIR`

Bonus: Separate FPM and CLI `PHP_INI_SCAN_DIR` Values

Try sharing `/etc/passwd`!