Given this and the pro-active involvement of some of my colleagues from Namshi, I really wanted to see some sort of community raise in this desert.

So a couple weeks ago I opened a google group for the PHP User Group Dubai – pretty empty right now – which is the place where we’re going to discuss meetings, ask for support and chillax in a nerdy way.

Yesterday we had our first meeting, where 7 of us discussed on the directions that we should initially take to gain the attention of others in the development community and which kind of formats to use during our meetings; the main points that came out of the meeting are:

• we will try to meet once a month
• we will try to set and prepare the next meeting before the beginning of Ramadan (~10 July)
• everyone will try to bring a new joiner (friend, colleague, etc.) to the next meeting
• we will have a talk and a discussion after that, most probably in front of a dinner accompanied by some sheesha
• most probably, since not everyone is aware of the potential and benefits of automated testing, we will prepare a talk on PHPUnit

So, if you are a PHP developer based in the UAE – not necessarily Dubai – why don’t you subscribe to our PHP User Group Dubai google group and introduce yourself? Do you want to understand why testing your applications manually it’s a waste of your energies and of money of your company? Do you know about Twig? Have you ever tried developing an application with AngularJS and some HTTP APIs as the backend of that app? How can you integrate the JWS authentication mechanism in your PHP code?

It would be great to see you at the next meeting!¹

Today I came up with a better solution that lets you take advantage of Twig full features without any hack (like the partial tag I was talking about in my previous post).

Instead of parsing the generated output as a string with Twig, we can store it into a template, which lets us use features like use, extends, include, thing that is almost impossible – in a clean way – if we use the Twig string loader:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?php

class Framework_Base_Controller
{
  public function render($templateName, array $parameters = array())
  {
      // ....
      // do stuff to render your template
      // we have the HTML output in the $templateOutput variable
      // ...

        $twig = new Twig_Environment(new Twig_Loader_String(), array(
          'autoescape' => false,
      ));

        return $twig->render($templateOutput, $parameters);
  }
}

Instead of using the Twig_String_Loader we would use an array loader, and store $templateOutput in a unique template, called __MAIN__.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

<?php

class Framework_Base_Controller
{
  public function render($templateName, array $parameters = array())
  {
      // ....
      // do stuff to render your template
      // we have the HTML output in the $templateOutput variable
      // ...

      $finder     = new Symfony\Component\Finder\Finder();
      $templates  = array();

      foreach ($finder->in('/path/to/twig/templates') as $file) {
          if (!$file->isDir()) {
              $templates[$file->getRelativePathName()] = $file->getContents();
          }
      }

      $loader = new Twig_Loader_Array($templates);
        $loader->setTemplate('__MAIN__', $templateOutput);

      $twig = new Twig_Environment($loader, array(
          'autoescape' => false,
      ));

      try {
          return $this->twig->render($templateName, Alice_Component_Registry::getAll());
      } catch (Twig_Error_Loader $e) {
          return $this->twig->render("__MAIN__", Alice_Component_Registry::getAll());
      }
  }
}

And that’s it!

Now you can write your own $templateName:

1
2
3
4
5
6
7

{ % extends '__MAIN__' % }

{ % use 'whatever' % }

{ % block somewhat % }
  some content
{ % endblock % }

In this situations, when a complete replacement would cost too much to your organization, you can take advantage of a wild integration between this advanced template engine and your existing code.

This article is outdated! A better approach was described here:
http://integrating-twig-in-your-legacy-code-part-2-a-less-wild-approach/

Approach

The main idea is that you should anyway have a man function which outputs what is being rendered on the view, so that you can capture that output and parse it via Twig, something like a render function in your controllers:

1
2
3
4
5
6
7
8
9
10
11
12
13

<?php

class My_Controller extends Framework_Base_Controller
{
  public function indexAction()
  {
      // ....
      // do stuff
      // ...

      return $this->render('my_template', $parametersForTheView);
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?php

class Framework_Base_Controller
{
  public function render($templateName, array $parameters = array())
  {
      // ....
      // do stuff to render your template
      // ...

      return $templateOutput;

      // will become
      return $this->twig->render($templateOutput);
  }
}

At this point the only thing that you need is to inject the Twig engine into your base controller and parse the output of your legacy templates with Twig²:

1
2
3
4
5
6
7
8
9
10

<?php

// your actual rendering:
return $this->twig->render($templateOutput);

// which means:
return $this->twig->render('<html><head><title>Hello world</title>...</html>');

// so that you can actually write twig in your templates:
return $this->twig->render('<html><head><title>{ % block title % }Hello world{ % endblock % }</title>...</html>');

The ‘block’ tag in the example above is having a space between curly brackets and the percentage char since my blog engine (octopress) doesn’t allow those tags them in code blocks.
In all of the next examples you will see Twig tags written like that.

Rendering content via Twig

To integrate Twig in your application it it really a matter of a few minutes: first, you will have to download and move the library inside your codebase, then, thanks to the PSR-0 autoloading (here we will be using Symfony2’s autoloader, but you can use any PSR-0 compliant autoloader) you just need to include it and setup Twig’s own autoloader:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?php

require_once __DIR__.'/vendor/symfony/Symfony/Component/ClassLoader/UniversalClassLoader.php';

use Symfony\Component\ClassLoader\UniversalClassLoader;

$loader = new UniversalClassLoader();
$loader->register();

$loader->registerNamespaces(array(
    'Twig' => __DIR__ . '/vendor/twig/lib/',
));

require_once __DIR__ . '/vendor/twig/lib/Twig/Autoloader.php';
Twig_Autoloader::register();

At this point, let’s get back to our render function, which we will need to modify in order to include Twig:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?php

class Framework_Base_Controller
{
  public function render($templateName, array $parameters = array())
  {
      // ....
      // do stuff to render your template
      // we have the HTML output in the $templateOutput variable
      // ...

        $twig = new Twig_Environment(new Twig_Loader_String(), array(
          'autoescape' => false,
      ));

        return $twig->render($templateOutput, $parameters);
  }
}

At this point we would be already able to write Twig code inside our templates:

1
2
3
4
5
6
7
8

{ % set posts = registry.get('blog_post').findByUser($user.id) % }
<p>
  { % for post in posts % }
      ...
  { % else % }
      This user didn't write any post
  { % endfor % }
</p>

A new tag

Unfortunately, to support some kind of inheritance, which is one of the greatest features of Twig, the situation becomes a little bit trickier: first of all, we will need to add to the parsed HTML some extra content to override blocks, then we will need to create a new Twig token parser in order to allow declaring multiple blocks with the same name, which is not allowed by the block tag.

Let’s say that all of your templates are including a base layout made of a very clean HTML structure:

1
2
3
4
5
6
7
8
9
10

<html>
  <head>
      <title><?php echo $title; ?></title>
      ...
      ...
  </head>
  <body>
      <?php echo $content; ?>
  </body>
</html>

At this point, since we are able to parse generated HTMLs with Twig, you can simply add a couple blocks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<html>
  <head>
      <title>
          { % block title % }<?php echo $title; ?>{ % endblock % }
      </title>
      ...
      ...
  </head>
  <body>
      { % block content % }
          <?php echo $content; ?>
      { % endblock% }
  </body>
</html>

After we do it, how can we override these blocks differently from each controllers’ actions? You simply include other Twig content at the end of the generated HTML:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<?php

class Framework_Base_Controller
{
  public function render($templateName, array $parameters = array())
  {
      // ....
      // do stuff to render your template
      // we have the HTML output in the $templateOutput variable
      // ...

        $twig = new Twig_Environment(new Twig_Loader_String(), array(
          'autoescape' => false,
      ));
      $twigTemplate = sprintf("path/to/twig/templates/%s.twig", $templateName;

      if (file_exists($twigTemplate))) {
          $templateOutput .= file_get_contents($twigTemplate);
      }

        return $twig->render($templateOutput, $parameters);
  }
}

And then override the content with your own twig template:

1
2
3

{ % block title % }
  About: this is our about page
{ % endblock % }

After you setup everything, you will realize that there is a huge problem here: since Twig doesn’t allow to declare blocks Twig, you can use the block tag!

To overcome the problem, you can simply add a new tag, partial:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

<?php

/**
 * Token parser for the twig engine that adds support to redefinable blocks,
 * under the 'partial' alias.
 * 
 * IE:
 * { % partial myPartial % }First{ % endpartial %}
 * { % partial myPartial % }Second{ % endpartial %}
 * 
 * will render "Second".
 * 
 * This is needed since the { % block % } tag doesnt support redefining blocks
 * with the string loader, it just supports it via inheritance.
 */
class PartialTokenParser extends Twig_TokenParser_Block
{
    /**
     * Parses the twig token in order to replace the 'partial' block.
     * 
     * @param Twig_Token $token
     * @return Twig_Node_BlockReference
     * @throws Twig_Error_Syntax
     */
    public function parse(Twig_Token $token)
    {
        $lineno = $token->getLine();
        $stream = $this->parser->getStream();
        $name = $stream->expect(Twig_Token::NAME_TYPE)->getValue();
        $this->parser->setBlock($name, $block = new Twig_Node_Block($name, new Twig_Node(array()), $lineno));
        $this->parser->pushLocalScope();
        $this->parser->pushBlockStack($name);

        if ($stream->test(Twig_Token::BLOCK_END_TYPE)) {
            $stream->next();

            $body = $this->parser->subparse(array($this, 'decideBlockEnd'), true);
            if ($stream->test(Twig_Token::NAME_TYPE)) {
                $value = $stream->next()->getValue();

                if ($value != $name) {
                    throw new Twig_Error_Syntax(sprintf("Expected endblock for block '$name' (but %s given)", $value), $stream->getCurrent()->getLine(), $stream->getFilename());
                }
            }
        } else {
            $body = new Twig_Node(array(
                new Twig_Node_Print($this->parser->getExpressionParser()->parseExpression(), $lineno),
            ));
        }
        $stream->expect(Twig_Token::BLOCK_END_TYPE);

        $block->setNode('body', $body);
        $this->parser->popBlockStack();
        $this->parser->popLocalScope();

        return new Twig_Node_BlockReference($name, $lineno, $this->getTag());
    }

    /**
     * Returns the tag this parses will look for.
     * 
     * @return string
     */
    public function getTag()
    {
        return 'partial';
    }

    /**
     * Decides when to stop parsing for an open 'partial' tag.
     * 
     * @param Twig_Token $token
     * @return bool
     */
    public function decideBlockEnd(Twig_Token $token)
    {
        return $token->test('endpartial');
    }
}

Then you just need to tell the Twig environment to add this token parser:

1
2
3
4
5
6

<?php

$twig = new Twig_Environment(new Twig_Loader_String(), array(
  'autoescape' => false,
));
$twig->addTokenParser(new PartialTokenParser());

and at this point you can use it in your templates:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<html>
  <head>
      <title>
          { % partial title % }<?php echo $title; ?>{ % endpartial % }
      </title>
      ...
      ...
  </head>
  <body>
      { % partial content % }
          <?php echo $content; ?>
      { % endpartial% }
  </body>
</html>

1
2
3

{ % partial title % }
  About: this is our about page
{ % endpartial % }

So?

If you spot any typo / mistake please do let me know: I wrote the example code adapting the one I had from a previous project so it might be that something slipped my mind.

Since I never dug that deep into Twig it might be that some things could be done in a cleaner way, so if you have suggestions or feedbacks I would strongly encourage you to go berserk mode in the comments section below.

This article gives an overview of the constraints and preferences that we wanted to implement in our SOA, which is mainly done with Symfony2, but most of it can be read in a framework/language-agnostic key.

Problems

We can at least identify 3 problems which pop up after you decide to layer your architecture and avoid a monolithic approach:

• given that every of your service will require some time (1~5 minutes) to be deployed, how do you ensure that you can release a new version of a service without the need of updating all the other services? If you have, for example, 9 machines and 3 services (A, B and C, 3 machines for each service), you cant really afford to deploy everywhere when you need to update just the service A, because you might need to shutdown service B and C during the deployment, while they dont really need to be updated. The solution here would be to update just a bunch of your servers

• how do you create Pull Requests and organize your repositories? This is not a trivial question: if you need a feature that involves changes in services A and B, and you have 2 repositories you will need to add some overhead on top of every single operation that you would usually do with a new feature

• is your software able to automatically support SOA? By this I mean, when you want to add a new service, how easy is to configure your architecture to be able to support the new layer? Of course, you would need something that lets you do this in a matter of a minute

Deployments

As I stated earlier, the solution is to be able to specify, upon deployments, which services need to be updated, and your best friend, here, could be something like Capistrano.

If you ever worked with capistrano, you know that deployments basically depend on the deploy.rb file, in which you can configure different stages of your architecture:

1
2
3

set :stages, %w(live staging)
set :default_stage, "staging"
require 'capistrano/ext/multistage'

In the example, weare declaring that our application can be deployed on 2 stages, live and staging; by doing a cap live deploy or cap deploy you are ready to either deploy to your live servers or staging ones, after configuring the staging files (live.rb and staging.rb):

1
2
3
4
5
6

role :web,        "company.com"
role :app,        "company.com", :primary => true
role :db,         "company.com", :primary => true

set :app_environment, "live"
set :deploy_to,   "/var/www/htdocs/#{application}.#{app_environment}"

This still doesn’t solve the problem of just deploying a single service, but to overcome it, thanks to the capistrano multistage extension, it’s a matter of configuring a few deployment files.

For example, here’s how you would write your deployment files once you have a couple services (A and B):

1
2
3

set :stages, %w(a-live a-staging b-live b-staging, live, staging)
set :default_stage, "staging"
require 'capistrano/ext/multistage'

1
2
3
4
5
6

role :web,        "a.company.com"
role :app,        "a.company.com", :primary => true
role :db,         "a.company.com", :primary => true

set :app_environment, "live"
set :deploy_to,   "/var/www/htdocs/#{application}.#{app_environment}"

As you see, we are creating a few different stages:

• serviceName-environment (ie. a-live), which includes servers for a specific environment of a service
• environment (ie. live), which includes the entire architecture, useful in those cases when you really want to deploy the entire architecture¹

For Symfony2, to ease your job, you can use capifony which takes care of configure the remaining, specific, crucial parts of the deployment for the framework (such as clearing and warming up the cache, installing the dependencies via composer and so on).

Software lifecycle

Ah, good old SCM problems!

Let’s say your are working on a traditional, monolithic application: you branch to implement a new feature, commit, push, open a pull request, the PR gets merged, deployed on staging, tested and then goes to live; very simple as well as efficient: zero overhead.

You might think that in SOAs this is not different: you cd into a specific’s service repository, branch to implement a new feature, commit, push, open a pull request, the PR gets merged in that repository, the service gets deployed on its staging servers, tested and then goes to live; very simple as well as efficient?

No, at all.

Thing is, often you will need to develop cross-service features, which require code to be updated in N different repositories: as a result, you will need to open N pull requests; if you think that this is not a problem, consider what happens once you pack and test everything and are ready to go live: a critical bug is found, and it couldn’t have been discovered earlier since it just happens due to some specific configuration that you have on the live environments: time to rollback N new services.

Since rollbacking is a very critical operation², you want to rollback each of the affected services separately, which makes your downtimes N times slower than a usual, monolithic rollback: in these cases, to be on the safe side, being able to rollback the entire architecture is a must.

That’s why I would advice to keep all of your services under one repositories, to avoid overheads:

• N projects in your IDE
• N SCM operations (git checkout -b myBranch as well as git push origin myBranch)
• N pull requests
• N code reviews

Due to this, honestly, I don’t really see the need of separating services into different repositories: when you deploy, you deploy a tag of the architecture itself (only on the servers which host the services to be updated with that tag), when you rollback, you rollback the entire architecture to a specific version.

This seems to go against what I preached earlier, while talking about deployments, but the truth is that you want to be on the safe side once something goes wrong: you can optimize deployments so that you can just deploy some services, but in case of rollback, you need to take an immediate, “total” action to restore all of your services.

Consider the situation from a very similar perspective coming from a very, very different context: cultivate an healthy colony of newborns in nature – as opposed to maintaining an healthy architecture on the internets.

Exactly like the mom of newborn birds, when it comes to feed (update) them, you would give the weaker ones, in order to help them develop as healthy as their stronger brothers, the biggest meals; but when it comes to rescue (rollback) them from a predator, you would crave for having an option to move them all in one go, without the risk of moving them one by one, leaving the unluckiest ones defenseless against their own fate.

Configuring new services in Symfony2

This post has to come to an end dealing with Symfony2, since, in our experience, we have decided to go SOA with this framework: all in all, we found that due to the integration with capistrano and the concept of bundles, together with the ability to have per-bundle specific hostnames, this framework is pretty friendly towards the ideas and constraints that we want to implement in our SOA: what we’ve seen is no rocket science, and even the approach that we are using with Symfony2 is nothing extraordinary, but it helps maintaining a very clean and efficient workflow while developing a SOA.

As I said, for deploying you might want to use capifony, and when it comes to isolate services in just one repository we realized that a good solution would be to have one Symfony2 application and create a bundle for each service that we need.

Thanks for the capabilities of Symfony2’s routing mechanism, you can also bind a subdomain to a specific bundle; once you create the bundle, you can tell symfony that the routes of that specific bundle can be matched only if the subdomain of the application matches a particular string:

1
2
3
4

mycompany_service_a:
    resource: "@AcmeServiceABundle/Resources/config/routing.yml"
    prefix:   /
    host:     service-a.mycompany.com

1
2
3

mycompany_service_a_index:
    pattern:  /index/{whateverParameter}
    defaults: { _controller: AcmeServiceABundle:Default:index }

In this case, the route mycompany_service_a_index will only be matched when the URL is using the hostname service-a.mycompany.com: for example, http://mycompany.com/index/param won’t match it; this is pretty interesting since it gives you the flexibility to develop features in just once repository, on as many services as you want.

We are still heavily experimenting, but out of a few approaches – for example route prefixing – we decided to go on with the ones I explained here for cleanness, clarity, efficiency and security of your development cycles and architecture: if your experience suggests something different or you want to share doubts, feel free to abuse of the comments section, since I am very open and interested to discuss this topic.

Basically, the specification defines 4 entities:

• JWS, JSON Web Signature, a signed representation of data
• JWT, JSON Web Token, a representation of data (it differs from JWS as JWT is not signed)
• JWE, JSON Web Encryption, an encrypted JSON representation of data
• JWA, JSON Web Algorithms, a list of safe algorithms to be used with JWS and JWE

For the sake of basic knowledge, we will only have a look at JWS and JWT / JWE now: the specifications about these entities are quite extensive and not very straightforward, so for further details you should really give them a look.

JWT

Basically, the token (JWT) is the simplest structure that you will deal with while implementing JOSE in our architecture; it is a string representation of some data base64 encoded (other types of encoding might be applied, but this is not madatory): the JWT differs from raw base64-encoded data since it also includes informations about the encoding itself, in the token’s header; by concatenating the base64-encoded version of the token header and payload (the actual data) you obtain what the specification calls signature input, which will then be used to create the signature (JWS).

JWS and JWE

After the JWT comes the JWS, which is a signed representation of the JWT; it differs from the token just because of the signature; on an higher step of the ladder comes the JWE instead, which lets you encrypt the data in order to achieve an higher security level: the examples in the ietf draft show you how to create JWEs with a pair of private / public keys.

Use case: how to authenticate stateless AJAX calls?

One of the needs that you might have is to, from JavaScript, make authenticated HTTP calls to one of your webservices: since you don’t want to expose the WS credentials on the JS service (the credentials would be readable by any client) a good solution might be to generate a JWS with a private OpenSSL key in your webservice, store it into a cookie accessible to the JS service, which would execute those calls including that cookie¹, which you can then verify while authenticating the call.

This workflow is pretty easy to understand, but the actual implementation is more than tricky, since the specification is quite abundant – especially about encryption algorithms.

In PHP we can use at least 3 libraries: one of them, Akita_JOSE, is pretty old (since the last commit was more than 7 months ago) but is very understandable and quite easy to use; another one, gree/jose, has itw own package on packagist and can be easily installed via composer: from a fast look at the source code on GitHub it looks good, even though it needs the phpsec library to be able to work².

The third option, which is the one that I built in the last couple of hours, is namshi/jose, which is very, very easy to use³: it currently only supports the RSA algorithm with sha256 hashing, but I guess that implementing other algorithms is less than trivial.

For example, let’s see how you would generate the JWS to be stored in a cookie:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?php

use Namshi\JOSE\JWS;

if ($username == 'correctUsername' && $pass = 'ok') {
    $user = Db::loadUserByUsername($username);

    $jws  = new JWS('RS256');
    $jws->setPayload(array(
        'uid' => $user->getid(),
    ));

    $privateKey = openssl_pkey_get_private("file://path/to/private.key");
    $jws->sign($privateKey);
    setcookie('identity', $jws->getTokenString());
}

and then the apps that want to execute authenticated calls on behalf of the user by using this cookie just need to include it in these calls; the server will just need to verify that the JWS in the cookie is valid:

1
2
3
4
5
6
7
8
9
10
11
12

<?php

use Namshi\JOSE\JWS;

$jws        = JWS::load($_COOKIE['identity']);
$public_key = openssl_pkey_get_public("/path/to/public.key");

if ($jws->verify($public_key)) {
    $paylod = $jws->getPayload();

    echo sprintf("Hey, my JS app just did an action authenticated as user #%s", $payload['id']);
}

That’s it: far from being a stable library, this is more a proof of concept that we, an Namshi, would like to see developing in the next weeks / months.

As always, comments, rants or – even better – pull requests are more than welcome!

Believe it or not, all of the following factors matter first for the business, then for the developers.

Testing

We are firm believers in automated test practices, and providing a layer that integrates very easily with testing tools (such as Behat or PHPUnit) is a must for us.

Symfony2 is a testing-prone framework because:

• it is well decoupled, so unit testing becomes very easy since you can mock objects, isolate classes and inject stub dependencies very easily
• it provides a first layer for functional testing (with PHPUnit): being an HTTP-centric framework, it provides a base class that lets you simulate HTTP requests and examine the output; needless to say, these kind of tests are way faster than the ones that you would write with tools like Selenium, since they don’t have the overhead of testing with an actual browser
• there is a Behat extension that lets you integrate the framework with this behavioral testing tool

At the end, you can see how Symfony2 and the ecosystem around it provide the proper toolset to run unit, functional and behavioral tests. If you do care about testing, this is already a huge point: we can’t afford our developers to waste a huge portion of their time doing manual testing, and we don’t want to increase overhead to build a manual QA team; since we are a technology startup, we should take advantage of technology to automate expensive tasks that harm the business, like manual testing¹.

Debugging

I kind of had so much fun when, while still working with symfony 1.4, I saw people developing with frameworks like Yii or Zend Framework 1, beating their heads on their desks tying to understand which view to modify, var_dumping SQL queries to output them and so on: symfony 1.4 already had a very powerful debug toolbar that would present all of these informations in order to ease debugging.

Symfony2 goes beyond what we had before, providing a way more powerful, extensible toolbar and an integrated profiler.

Database inspection will let you realize how many queries you are running and see the SQL of all of them, with a nice overview of the time they take, while the profiler itself includes informations about every step of the application: for example, with Joomla, can you tell this easily how long it took to render a particular view and how much memory was used to execute a controller’s action?

Now, imagine each of your developers (let’s say you have a team of 6), spending 1 hour (very conservative estimate) out of 40 (a working week) trying to obtain informations that debugging tools natively give you: it’s almost a day per week; multiply that day for 52 weeks in a year and you will end up loosing one of your developers for two moths.

We all honestly can’t afford to let a guy leave the company for 2 months for free, so why would we keep using counterproductive tools?

Doctrine 2

It is no news that we, at Namshi, are working with a Service-Oriented Architecture, and we are highly benefiting from the easy integration that Symfony2 provides for Doctrine.

One of the rules of thumb of designing SOAs is that you can provide access to the same data source to different services: in simple terms, instead of talking via webservices or messaging queues, services can simply access the data stored somewhere by other ones.

Well, Doctrine 2 is the cherry on top of the cake to access that somewhere: natively providing support for multiple DB connections and object-relational mappings, you can safely use this tool, within Symfony2, to handle read and writes to different databases without polluting the domain model of each of the services that take advantage of Doctrine; in addition to this, I should enumerate the huge list of good things that working with a data mapper like Doctrine 2 brings on the table.

On another note, sharing the data model among different services helps you overcoming though situations where webservices or messaging queues are not enough: think about a service which, due to an update, needs to modify half a milion records that “belong” to another service; of course, istantly having 500k messages in a queue implies a long, very long time to process them, while a webservice might not be fully ok with sending a huge payload over the HTTP protocol – and, moreover, how do you start testing this feature, when your developers need to send a lot of MB through their browser? It is painful, believe me.

At this point, the ability of directly accessing different DBs come out as a swiss-army knife, as you can directly execute the 500k updates, in batch, from the original service.

Deployments

Symfony2 has an out of the box integration with Capistrano, the most popular automated deployment tool in the market.

This means that you should forget about wasting time, money and energy to develop your own in-house solution to automate deployments or, even worse, rely on manual procedure, which are prone to errors where it hurts the most, on the “live server”.

DIC

Let’s say that you, for example, are using Graylog2 to handle logs in your application: while you are developing locally, you won’t have a graylog2 server to connect to, since it might be that you want to keep your machine a bit cleaner and you might find more useful to read local logs from a file in the filesystem, or directly output them to the browser.

In Symfony2, thanks to the dependency-injection container, you can define the logger as a service:

1
2
3
4
5
6
7
8
9
10
11

logger:
    class: 'Monolog\Logger'
    arguments:
        name:           "applicationName-%kernel.environment%"
    calls:
        - [ pushHandler, [ @monolog_handler.graylog ] ]
monolog_handler.graylog:
    class: 'Monolog\Handler\GelfHandler'
    arguments:
        publisher: @gelf.message_publisher
        level:     200

and, for development environments, you can simply override the configuration in the config_dev.yml file:

1
2
3
4

monolog_handler.graylog:
    class: 'Monolog\Handler\StreamHandler'
    arguments:
      stream: "php://stdout"

This will allow you to output errors that would normally go to graylog2 directly to the developer’s browser, easing debugging when you can afford to display errors in the browser – thing that is not possible who’s viewing your application is a potential customer.

Apart from all the technicalities involved in using a DIC, I would like to focus on one point: again, simplicity and speed to implement a solution to a problem (having different log handlers depending on the application’s environment, in this case) are a winning factor for your development team, which is translated in more productivity for your company.

Bundles

When we kickstarted our first Symfony2-based service in our architecture, we decided to meld together 2 applications that support our CRM and ERP systems: being inside Symfony2, these layers are fully isolated in separate bundles, giving us the ability of phisically decoupling them in 2 installations in a matter of minutes.

Bundles are probably one of the most powerful concepts of Symfony2, since they are micro-applications inside your main application: being able to totally separate logics from different domains helps you in keeping a clean separation of concerns and autonomously develop every single feature of your domain.

Declarative code

Consider the following snippet, written using the Symfony2 Finder component:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?php

use Symfony\Component\Finder\Finder;
use Zend_Service_Amazon_S3 as Amazon_S3;

$s3 = new Amazon_S3($key, $secret);
$s3->registerStreamWrapper("s3");

$finder = new Finder();
$finder->name('photos*')->size('< 100K')->date('since 1 hour ago');

foreach ($finder->in('s3://bucket-name') as $file) {
    print $file->getFilename();
}

After your first look at this code, you already know what it is doing: now imagine that your team of developers need to, instead, try to understand how the Drupal framework works.

Taken from Drupal’s source code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?php

/**
 * Process comment_confirm_delete form submissions.
 */
function comment_confirm_delete_submit($form, &$form_state) {
  $comment = $form['#comment'];
  // Delete the comment and its replies.
  comment_delete($comment->cid);
  drupal_set_message(t('The comment and all its replies have been deleted.'));
  watchdog('content', 'Deleted comment @cid and its replies.', array('@cid' => $comment->cid));
  // Clear the cache so an anonymous user sees that his comment was deleted.
  cache_clear_all();

  $form_state['redirect'] = "node/$comment->nid";
}

I’m far from saying that Drupal sucks, but some questions rise into my mind:

• why do I have a $form and a separate $form_state?
• what is watchdog() doing? Is it used for logging? Or to display flash messages?
• what is a cid? And what about the nid?
• why should I clear my entire application’s cache to notify a user of a change?

See, you don’t want your developers to have to go through an entire application to understand what a piece of code does.

Best practices

Symfony2 is a framework made to take advantage of clean and clear tested patterns as well as tools to improve the final developer’s productivity: imagine your team, working six months on this framework; how much would they learn? How many structural changes would they be able to do on your application without introducing regressions?

For startups, by the way, a huge plus comes from the fact that being highly decoupled, Symfony2 helps when you want to drastically replace a piece of software, or an adapter, with another one: for example, thanks to the dependency-injection container, you would be able to replace application services with others that have the same API, but a different implementation.

It is clear enough that Symfony2 provides the flexibility you need to reach a very short time to market and increases your developers’ awareness and efficiency by giving them the guidance and the tools they need to care about the domain of your services and not about how many bugs they would introduce by changing an untested piece of code.

All in all…is it Symfony2?

A very simple question that you should ask yourself at the end of this reading is: but, all in all, is this all thanks to Symfony2 or its surrounding environment?

It is its surrounding environment, which was born thanks to the framework itself: when Symfony2 was released, no other framework had the same level of quality that the open source product from SensioLabs could offer; a natural effect of this was that the majority of well-known open source PHP developers got amused by this framework and embraced its way.

Basically, Symfony2 is a framework chosen by the community, thus it can take advantage of all the efforts of the OS developers around it: from automated deployment tools to fully integrated ORMs, from testing frameworks to tutorials and best practices, through native, advanced debugging tools, Symfony2 is, as of today the most complete framework available in the PHP ecosystem when you take in consideration learning curve, integrations, stability and performances (don’t forget that one of top 100 website in the Alexa rank is powered by Symfony2²).

Can other frameworks do all of this?

For the benefit of your business, this is the main question that you should ask yourself.

Automated testing is great idea

If you have a ticket which requires a lot of changes (20 files, for example) Automated tests are very helpful and save your time, it saves about 30% of your time; even if it’s still not perfect, its good
For example look at this PR [link to a PR on Github], if i want to test all cases i need about half a day but it only took half an hour

The good Marco Pivetta took initiative in adding proxy generation for hydrators, and in a lonely branch he’s pretty far with the progress on the matter: his latest tests show that the hydration process is improved by 80% which is a very good news, considered metadata-based software usually need to find a way to drastically improve performances because of the basic lack of performances due to having to read metadata.

Hoping to see this changes integrated in a stable branch very soon!

Luckily, this was a great opportunity to meet some good old friends and the talk went very well: of course, the reharsal at the PHP User Group in Rome earlier this week was a bless.

It basically deals with the phases that I experienced with our team in Namshi, with some spicy details on recruiting (from looking for talents to conducting interviews), how to change processes to help people improving (and not change people to improve processes) and a big picture on what we were able to plan and achieve in this year together, from a technical point of view.

At the end of the talk I invited everyone interested, and I will repeat myself here, to look for the open positions on our careers site, since we are currently hiring a Lead Developer and a PHP Developer (a junior or an intermediate, it’s more about the approach rather than the hands-on experience).

Here are the slides of the talk, in a slightly better (and more colored) version compared to the ones I used in Rome:

Fun thing is, since I can’t miss the opportunity of getting back in touch with a few people, I will be joining 4 tech events:

• Codemotion – already happened, I was there on the 23rd – the conference open to all the programming languages, in Rome
• PHP User group Rome, a meetup that is held monthly, for one of the user groups that I helped launching here in Italy, again in Rome, where I gave a talk about assembling a mid-sized PHP team
• PHP.TO.START – happening right now – a yearly event for the north-western italian PHP community, where I’ll be presenting the same talk I had at the PUG Rome
• PHP User Group Friuli, another meetup organized by one of the italian PHP User Groups, where I’ll be presenting an unknown topic (at least for now)

An important thing: I am actively recruiting for our company, Namshi, so if you are interested in one of the open positions that we have right now (in particular, Lead Developer and PHP Developer), feel free to approach me if you find me at the upcoming events (in Turin and Udine).

But to aim expansion, one needs to go beyond those shared layers and start customizing his products and services, and in terms of software development nothing can help you more than service-oriented architectures, or SOA.

So, what’s the goal of this post? Basically providing our view on how we are going to shift from our current architecture, which is already a composite, to a more powerful layer of services.

Identifying the service

One of the first steps in order to dig into the implementation is to actually identity a first bunch of functionalities that should be incorporated as standalone services.

Usually, opportunities for new services pop up when it’s time to introduce a new functionality or the cost of fixing / implementation of a change request are too high: for example, if you want to add the ability to send SMSes from your website, a good service would be one which just deals with the receiving an input event, assembling a message and contacting the real SMS provider via webservice in order to dispatch the message; another good example is identity: if you are struggling with different userbases that need to be in sync, a good solution would be to centralize identities and provide a service which does, at least, authentication.

Data

Another tipical question is how to manage and organize data when you have a de-centralized architecture.

In SOA terms, usually data is shared among the services but this doesnt mean that each service can’t have its own data-layer: it is often seen a very old fashioned RDBMS shared across all the services and some of them using a less traditional solution, like a NoSQL DB; this is mainly done to achieve better performances and different data-retrieval patterns

Think about legacy applications that have a model which can be extensively customized by the end user, that usually implement the EAV pattern, getting stuck into performance bottlenecks, while a document-db like MongoDB or CouchDB would perfectly solve the issue.

If you are running, for example, an e-commerce system, you may want to have transactions and identities in a solid and robust system like MSSQL, while your frontend can actually run with MongoDB: once the user purchases a product, via webservice you store it into MSSQL.

Services need

One typical aspect, in SOA terms, is seeking answers to our questions (read getting responses for our requests), a problem which we can overcame with a simple solution: when a service needs another one, we talk about APIs.

For example, your frontend might offer authentication, while the Identity manager is a service providing identities to multiple layers of your architecture: when the frontend needs to authenticate a user, it will directly rely on the Identity service, asking him to authenticate the user with the credentials he or she submitted to the frontend.

APIs can be traditionally categorized into a few types:

• mess: “messy” API don’t follow structured rules (it cab be plain-old XML over HTTP or a replication of DB writes and reads in JSON format); they can be very useful when you need to kickstart a new, small and simple API
• HTTP API: services that semantically expose their domain model in terms of resources, embracing the HTTP specification
• REST: hypermedia services
• SOAP: services using strict interfaces between clients and servers, following the SOAP spec

No matter what, you will always find yourself dealing with APIs if you decided to go for SOA: it is the simplest way to provide data-exchange mechanisms to layers that don’t fully know each other’s domain.

Services listen

Another very common scenario, is when services “listen”, waiting for notifications sent across by other components of the architecture: you are probably already thinking about messaging queues and message notifications, and you are right.

A event-driven process can be achieved when we have tools such as RabbitMQ helping in gathering and dispatching notifications to various parts of the architecture: with Rabbit, a service can dispatch a message to a queue and another one (or ones), through a daemon, consumes the message.

Thinking about what I mentioned earlier, an SMS-dispatching mechanism could fit in this context really well: think about SMSes that are sent once the user completes certain actions on your frontend (by gaining credits, placing an order on your e-commerce or so on); once the user completes an action, a notification will be sent out and whoever needs to listen to that message will catch and process it.

So far so good

In our fast and new journey towards integrating services into our architecture, we are finding ourselves pretty well: it is no news that we are using RabbitMQ and Symfony2 for our new, isolated services, and that we already identified a few services that can run on their own, decoupled context.

Thinking in SOA terms, by the way, brings out a new set of problems, like thinking in terms of architecture, and not of application: you don’t deploy a new version of your application, you update a part of the architecture; your system is decoupled, from the code to the processes you use to handle them. And what about the complications in the development environments? And which monitoring tool should I use to understand that all the components are working alltogether? And…

There’s room for generic problems that everyone faced and that we will face as well, and I think it will be very interesting to share our approach and the vision we had in our own context.

I guess the error is pretty common, and it’s really easy to fix, since the problem is that the binary is not executable:

1

chmod +x app/console

…and you’re done.

Today I am going to show the approach that we just kickstarted, at Namshi, in order to help designers and developers testing frontend changes in a more automated, thus easier, way.

Thanks to the NamshiVoyeurBundle, it is really easy to start increasing the efficiency of your testing department, even if coders do not want to write automated tests.

The bundle, that you can use inside a Symfony2 application, is actually very small and can be extrapolated to be integrated in other frameworks (like ZF2 or Cake).

The idea is very simple: you take some screenshots of a website, deploy a new version, take another set of screenshots (at the same URLs) and then compare them, generating an image diff.

After you install the NamshiVoyeurBundle (via composer), it is really easy to start taking screenshots; you just have to configure a few services and some parameters:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

parameters:
    namshi_voyeur:
      browsers:
        - firefox
        - safari
        - chrome
      urls:
        homepage:     "/"
        new-arrivals: "new-products"
        women:        "women-shoes"
      shots_dir: "/Users/you/Downloads/screenshots"
      base_url:       "http://en-ae.namshi.com/"

services:
    safari:
        class:  Behat\Mink\Driver\Selenium2Driver
        calls:
          - [start]
        arguments:
          browser: safari
    firefox:
        class:  Behat\Mink\Driver\Selenium2Driver
        calls:
          - [start]
    chrome:
        class:  Behat\Mink\Driver\Selenium2Driver
        calls:
          - [start]
        arguments:
          browser: chrome

This configuration basically tells Voyeur that you will be taking screenshots of three URLs:

• http://en-ae.namshi.com/
• http://en-ae.namshi.com/new-products
• http://en-ae.namshi.com/women-shoes

with safari, firefox and google chrome.

To run the Voyeur, use the cli:

1

php app/console namshi:voyeur

Screenshots will be saved at /Users/you/Downloads/screenshots.

At this point, after you deployed a new version of the code, run the Voyeur again, and you will be reay to generate the diffs between the screenshots:

1

php app/console namshi:voyeur:diff /Users/you/Downloads/screenshots/firefox/2013/03/10/1200 /Users/you/Downloads/screenshots/firefox/2013/03/10/1205

Diffs will be generated at /Users/you/Downloads/screenshots/firefox/2013/03/10/1205/diff.

That’s it: now you can start having a look at what changed and ask your developers to do the same, even on their local machine, before committing any changes.

We are in the process of building new applications and API to move towards a SOA architecture, using cutting-edge libraries (like Symfony2), tools (like Behat and Composer) and practices.

The Job Description is still on his way but I can post here some of the competencies that we are looking forward to warmly welcome in the crazy Namshi family:

• enterprise practices and patterns, such as automated testing, pair programming
• HTTP webservices
• you buzz around the PHP5.3+ ecosystem, from the Symfony2 components to tools such as Composer
• you have Git for breakfast
• RDBMS (MySQL, Percona)
• can deal with pre-PHP5.3 frameworks (Zend 1 and Yii)
• Solr and/or Memcached
• passionate about open source and ideally contributing to it.

The salary reflects the demand of skills that we are looking for, and you will be responsible of the day-to-day development of our applications.

Just to mention: the United Arab Emirated are a tax-free country, the weather is amazing (even though for a couple months, during the summer, its really hot) and it’s an all-around country, meaning that if you want, you can enjoy the glamorous side of Dubai or, if you dont feel like¹, you can head towards the old town and enjoy the cultural meshup made of Indonesia, Thai and Indian food or Arab heritage.

I rushed this post, I know, so you can ask for more informations in the comments or, even better, drop me an email at alex.nadalin@namshi.com.

Cheers!

Basically, it happens when you serialize an object of class A and try to unserialize it when class A doesn’t exist anymore.

How (the hell) do I get there?

If you are working with auto-generated proxy classes, store objects in the session and then clear your cache, once you retrieve an object from the session you are going to face it. A solution is to re-generate all the proxies before retrieving objects from the session¹.

Profiling in a few words

As PHP developers, we are rarely used to profiling: essentially, most of our applications are not bound to extensive CPU usage or insanely huge data-processing operations; the scope of the language is very clear and even though we might need to profile, once in a while, it’s unlikely that we will end up having problems like optimizing MapReduce algorhitms.

But sometimes we do need to profile, and this will bring on the table bottlenecks of your applications: within a session of inspection, you will likely find optimizations that would lead to a 20/30% faster execution time, by just changing your backend (PHP) code¹.

Why Webgrind

Among all the available profilers for PHP, I choose to go with Webgrind for a bunch of reasons:

• nowadays, I am mostly developing on a Mac², so KCacheGrind wasn’t an option
• I didn’t want to install XHPROF as it usually takes a few minutes, even though is probably the best profiler for PHP: facebook uses it in production, and it’s able to generate a lot of reports that would make you face performance optimizations from various perspectives
• Webgrind offers a zero-setup installation

Installation with XDebug

XDebug is a must for profiling, as it’s the tool through which we can generate the Cachegrind files, that are basically reports on the costs of your application’s calls.

To enable XDebug’s profiling you will have to tweak your php.ini’s configuration:

1

xdebug.profiler_enable = 1

Beware that profiling each request your application processes can be an expensive job (pages that would usually load in 2/3 seconds can take up to 10 seconds), so you should – instead of enabling the profiler by default – activate the enable_trigger directive, which will make XDebug profile your application only if a specific GET or POST parameter is specified within the request:

1
2

xdebug.profiler_enable = 0
xdebug.profiler_enable_trigger = 1

Dont forget to restart apache once you made the changes:

1

sudo apachectl restart

By visiting your application and specifying a special GET parameter in the URL, you will run your first profiled PHP response: supposing that you want to profile the code that runs http://dev.project.com, just visit http://dev.project.com?XDEBUG_PROFILE=true

Once you’re done with the XDebug configuration, it’s time to install Webgrind:

1
2
3

cd /path/to/your/home/projects

git clone git://github.com/jokkedk/webgrind.git

That’s it!

You can now access Webgrind at 127.0.0.1/webgrind or – if you prefer – set up a virtual host for it:

1
2
3
4
5
6
7
8
9
10
11
12

<VirtualHost *:80>
    DocumentRoot "/path/to/your/home/projects/webgrind"

    ServerName webgrind

    <Directory "/path/to/your/home/projects/webgrind">
        Options Indexes FollowSymLinks MultiViews
        AllowOverride all
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

and have Webgrind running at http://webgrind/.

Looking at the results

Once your application runs, XDebug will generate the cachegrind files that Webgrind will analyze: after each PHP response is served from your application, you can inspect the results from the Webgrind interface, by just selecting the first file of the list: it might take some time for Webgrind to generate the first report, as cachegrind files can easily size up to 100/200 megabytes (files below ~50MB will be read in 10 seconds or so).

When the report is generated, you will see the results: I strongly recommend to generate a report in milliseconds, as it will give you a direct overview on how much time a function takes rather than having this value as a percentage compared to the entire application’s run.

If you order results by Total inclusive cost, you will exactly see which ones are the most expensive functions of your applications: in the example, you will see that the Doctrine\ODM\OrientDB\Mapper::hydrate method really kills the performances of my application (10.6 seconds).

Having this kind of report is not really useful, as usually you need to dig deeper to understand which exact step is making that function taking all that time: you can investigate further by clicking on a function, action that will open the call stack after that function is called:

as you see, the problems, here, lies in Doctrine\ODM\OrientDB\Mapper::createDocument (6.2 seconds) and Doctrine\ODM\OrientDB\Mapper::findClassMappingInDirectories (4.3 seconds), so there you have the explanation why Doctrine\ODM\OrientDB\Mapper::hydrate takes more than 10 seconds.

Then, take your time to investigate even further and make the optimal changes in your application, run it with the profiler enabled once more and have a look at the results:

As you see, after I tweaked my code, Doctrine\ODM\OrientDB\Mapper::hydrate is not even the most expensive function at all (Sharah\Controller::getPartial is), and the previously performance-killer methods, which would take ~6 and ~4 seconds, are now respectively taking ~1 and ~0.1 seconds.

The call graph

Another interesting feature of Webgrind³ is the ability to generate a call graph to visualize bottlenecks in the application: by having a look at the graph you will have a top-down overview on how much execution time (expressed in percentage) a function will take.

When you look at it, you should question every step of the graph and ask yourself is that specific function should really take that amount of time.

For example:

• if a controller takes 20% of the time to run (called TTR from now on), it might be that you have a design flaw, as it should be the most expensive part of your application, calling the models and rendering the view (which are included in the calculated TTR)
• if a model’s method is taking 60% TTR, there is a bad smell: how come that just retrieving data once is taking more than half of the TTR?
• if bootstrapping the application takes 15% TTR, then it’s fine, as that is usually the time a well-abstracted framework needs to provide you a solid foundation to develop on top of

In the image above, you will see that 87% of the execution time is taken by the controller’s action (which is fine) and then equally distributed (10/20%) across various other functions that controller calls.

Conclusions

This is not an extensive guide on profiling, neither a fair comparison of PHP profilers (as I said, I picked Webgrind, last night, more because of the stress-free installation rather than its actual capabilities⁴), but I hope it can give you a good quickstart guide to start optimizing bottlenecks in your applications.

One thing that I would really like to point out is to stop useless optimizations: there is no need to “drop double quotes in favour of single quotes” because those are such small optimizations that you will never feel in your application; create useful benchmarks and always question your choices: running a solid benchmark and profiling your application properly will also tell you that sometimes the best design is probably unpractical to perform really well.

Prelude

Premature optimization is the root of all evil

In these days I was testing performances of a service I am building with OrientDB and the doctrine ODM that we built so far.

Following one of the golden rules for software architects, we didn’t paid attention to performances – at first – but rather went for a design which would allow us to inject behaviours and easily change portions of code (also thanks to the test suite), I knew that I would have noticed huge flaws at performance level when testing it with production data.

A few days ago we committed fetchplans for repository classes, but it wasnt enough: rather then concentrating on which data we should fetch, I realized one major improvement could be applied on how we map data.

The golden rule

When you write a OXM (object-something mapper) you will shortly understand that a huge portion of your job consist into abstract your design, to ease integration of multiple components into your application: repositories, the object manager, POXO, the data mapper, proxy classes and so on.

Of course, abstraction comes with a cost: slow performances, so one of the first things that you do is starting to cache everything.

How we did it

With a single commit – there’s always room for improvements – the ODM is now able to hydrate objects 5 times faster: when you hydrate similar objects from OrientDB (for example, 2 records that share the same attributes’ values, like is_published or country), there is no need to duplicate operations, so we added a cached inflector (with an in-memory / single request cache) and did some other improvements to the Mapper:

• cached the relations between PHP classes and OrientDB classes (if 2 records of the same OrientDB class are hydrated, there is only one single search operation to find the PHP class that should map them)
• cached the casting of properties (if 2 objects have the same value for the is_published attribute, casting is done once)
• cached properties’ annotations (property-level annotations are inspected once per class)

There is no rocket science in what we did, but benchmarks ensure that it’s a huge performance improvement.

By the way, we used Webgrind

Doing almost all of my work from a Mac, I kind of missed KCacheGrind for profiling, so I was looking for an alternative (no, installing XHPROF isn’t an alternative at 2 in the morning) and I found Webgrind (which is cross-platform), a web profiler that requires zero setup: you basically just need to provide it access from the webserver and, by opening it with a browser, the application automatically launches and parses the cachegrind files generated by XDebug.

Webgrind’s code is a bit of a mess, but then, the result is still pretty good – you get a good overview of the expensiveness of your calls as well as a call graph compiled in DOT, which is a de-facto standard for graph generation.

Prelude

In the last few weeks I’ve started working, for fun and profit, to a personal project, nothing really exciting as of now.

The thing is, since I wanted to get back on some cool piece of software, I decided to go for OrientDB for the persistence and a mini-framework a-la Symfony2 as foundation for the PHP application – I actually considered NodeJS first, but I need a prototype in 2 months so…

Point being, I’d like to share with you my basic approach to the OrientDB ODM.

The container

Given I’ve been inspired to Symfony2, instantiating the main ODM classes happens in the DIC:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

services:
  orientdb.binding.parameters:
    class: Doctrine\OrientDB\Binding\BindingParameters
    arguments:
      host:     127.0.0.1
      port:     2480
      username: admin
      password: admin
      database: DBNAME
  orientdb.binding:
    class: Doctrine\OrientDB\Binding\HttpBinding
    arguments:
      parameters: @orientdb.binding.parameters
  odm:
    class: Doctrine\ODM\OrientDB\Manager
    arguments:
      mapper: @odm.mapper
      binding: @orientdb.binding
  odm.mapper:
    class: Doctrine\ODM\OrientDB\Mapper
    arguments:
      documentProxyDirectory: %base-dir%/tmp/
      annotationReader: @odm.annotation-reader
    calls:
      - [setDocumentDirectories, [ %base-dir%/src/PROJECT/Entity/ : "PROJECT\Entity" ] ]
  odm.annotation-reader:
    class: Doctrine\ODM\OrientDB\Mapper\Annotations\Reader
    arguments:
      cacheReader: @cache.array
  cache.array:
    class: Doctrine\Common\Cache\ArrayCache

parameters:
  base-dir: /Users/odino/Sites/PROJECT

As you see, you need:

• the Manager, which requires a Mapper and a connection to OrientDB through a binding class implementing the Doctrine\OrientDB\Binding\BindingInterface
• the Mapper, which requires a directory where it can write proxy classes (for lazy loading), and an annotation reader (this is not required, I’ll explain it later), plus a source directory to locate entities
• the HttpBinding, used by the Manager, that does raw queries to the OrientDB server
• the Annotations\Reader
• a cache implementing the interface Doctrine\Common\Cache\Cache: in dev environments it is needed since ApcCache is the default one, and you would need to flush APC every time you change an annotation in your entities (we will probably change it and put ArrayCache by default, so that you will need to tweak the live environment, not the dev one)

Autoloading

The autoloading is straightforward, thanks to the PSR-0; the only thing that you should keep in mind is that you will need to specify a separate autoloader for proxy classes, since they can be generated wherever you want (ideally, in a temporary folder, since they should be removed every time you deploy):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<?php

require_once __DIR__.'/../vendor/symfony/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php';

use Symfony\Component\ClassLoader\UniversalClassLoader;

$loader = new UniversalClassLoader();

$loader->registerNamespaces(array(
    'Symfony'                       => __DIR__.'/../vendor/symfony/symfony/src',
    'Doctrine\Common'               => __DIR__.'/../vendor/doctrine/common/lib',
    'Doctrine\OrientDB'             => __DIR__.'/../vendor/doctrine/orientdb-odm/src',
    'Doctrine\ODM\OrientDB'         => __DIR__.'/../vendor/doctrine/orientdb-odm/src',
    'Doctrine\OrientDB\Proxy'       => __DIR__.'/../tmp',
));

$loader->register();

You should set the autoloader for Doctrine\OrientDB\Proxy accordingly to the argument documentProxyDirectory of the odm.mapper service.

Entities

Following what we specified in the container.yml, entities should be located in %base-dir%/src/PROJECT/Entity/ and follow the namespace PROJECT\Entity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

<?php

namespace PROJECT\Entity;

use Doctrine\ODM\OrientDB\Mapper\Annotations as ODM;

/**
* @ODM\Document(class="user")
*/
class User
{
    /**
     * @ODM\Property(name="@rid", type="string")
     */
    protected $rid;

    /**
     * @ODM\Property(type="string")
     */
    protected $email;

    /**
     * @ODM\Property(type="string", notnull="false")
     */
    protected $nick;

    /**
     * @ODM\Property(type="linklist")
     */
    protected $addresses;

    /**
     * Returns the nickname of the user, or his email if he has no nick set.
     * 
     * @return string
     */
    public function getNick()
    {
        return $this->nick ?: $this->getEmail();
    }

    public function setNick($nick)
    {
        $this->nick = $nick;
    }

    public function getEmail()
    {
        return $this->email;
    }

    public function setEmail($email)
    {
        $this->email = $email;
    }

    public function getAddresses()
    {
        return $this->addresses;
    }

    public function setAddresses($adresses)
    {
        $this->addresses = $addresses;
    }

    public function getRid()
    {
        return $this->rid;
    }

    public function setRid($rid)
    {
        $this->rid = $rid;
    }
}

As you see, mapping an entity is pretty easy: the first annotation is at class level, to define which OrientDB classes are mapped by the entity, then for every property that you want to be persisted / hydrated, you define another annotation and public getters / setters; if you want the property to be public, you dont need getters / setters.

The property-level annotation has 3 parameters:

• type: defines the type of the property in OrientDB ( boolean, link, linklist, string, integer, etc )
• name: the name of the attribute in the OrientDB class (you might have a PHP property called $createdAt and in OrientDB you call it created_at)
• notnull: defines whether the property can be null or not¹

What about controllers?

You can access the ODM from within controllers of your application by just using the container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?php

namespace PROJECT\Controller;

use Project\Entity\User;

class UserController
{
  public function somethingAction()
  {
      $user       = new User();
      $manager    = $this->getService('odm');

      $manager->...
  }
}

Repositories

At this point, after boostrapping the environment and creating your first entity, you might want to play with the repository in your controllers, to manipulate and retrieve collections:

1
2
3
4

<?php

$manager      = $this->getService('odm');
$userRepository = $manager->getRepository('PROJECT\Entity\User')

then, with the repository, you can start retrieving objects:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<?php

// find all users
$userRepository->findAll();

// find one user given its RID
$userRepository->find($rid);

// find all users with the nick "overlord"
$userRepository->findByNick("overlord");

// find the user with the nick "TheOnlyOverlord"
$userRepository->findOneByNick("TheOnlyOverlord");

// find jack's wife
$jack  = $userRepository->findOneByName("Jack");
$wifey = $userRepository->findOneBySpouse($jack); // spouse is an attribute of type "link"

and it’s not over, since you can, of course, add custom repository classes.

Custom repositories must be located in the entity’s folder and follow the naming convention EntitytheymapRepository: for our User entity, we would need to create a UserRepository class in %base-dir%/src/PROJECT/Entity/:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?php

namespace PROJECT\Entity;

use Doctrine\ODM\OrientDB\Repository;

class UserRepository extends Repository
{
    /**
     * Retrieves a random user.
     * 
     * @return \PROJECT\Entity\User
     */
    public function findRandomUser()
    {
        return array_rand($this->findAll());
    }
}

so then you can call your new methods over repositories:

1
2
3

<?php

$manager->getRepository('PROJECT\Entity\User')->findRandomUser();

Can I haz raw queries?

Entities and repositories are good, but what about adding some SQL+² to the mix?

That’s very easy, thanks to the query builder that’s packed with the ODM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

<?php

use Doctrine\OrientDB\Query\Query;

// instantiate a query object
$query = new Query();

// simple SELECT
$query->from(array('user'))->where('nick = ?', $nick);

// throwing some spice into the mix
$query->orWhere('attribute = ?', $attribute)
    ->orWhere('(this IS NULL OR that IS NOT NULL)')
    ->limit(10)
    ->orderBy(...);

// SELECTing a single record
$query->from(array($rid));

// SELECTing two records
$query->from(array($rid1, $rid2));

When you manipulate the $query object you are basically creating an SQL query with an object-oriented fluent interface; to eventually execute the query, just pass the object to the Manager:

1
2
3
4
5
6

<?php

$query = new Query();
$query->from(array('user'))->where('gender = ?', "male");

$males = $manager->execute($query);

Point being, how do you save data?

Since persistence is not already handled by the Manager, you will need to use raw queries for now:

1
2
3
4
5
6
7
8
9
10

<?php

$user = array(
  'name' => 'Jack'
);

$query = new Query();
$query->insert()->into('user')->fields(array_keys($user))->values($user);

$manager->execute($query);

From the trenches

We’ve been very active since a couple months, and we’ve actually been able to roll out some major bugfixes and improvements (more than 10 in the last few weeks):

• repositories filtering by multiple criterias
• custom repository classes
• added ability to map timestamps as DateTime objects
• unable to update attributes if they are a collection
• support for INSERTing collections
• proxy classes dont import signatures
• findBy* and findOneBy* “magic” methods
• fetchplans in find* methods of repositories
• following SQL+, added REBUILD INDEX command

I would not advise you to install one of the old tags, or even the last one, which brings the namespace changes for the incubation in the Doctrine organization, but to install it directly from master via composer:

1

"doctrine/orientdb-odm": "dev-master",

as we are constantly doing bugfixes and so on (I would day you would get an update – at least – every week).

That is it, now start playing around!

New daddy

I’ve blogged so many times about an imminent integration into the Doctrine ecosystem, but due to the workload of our contributors and some major issues we wanted to solve before this, we were only able to seriously approach the Doctrine team today.

This is a very good news, as we will be able to take advantage of the experience of all the doctrine contributors as well as have a bigger stage where we can show the ODM: the biggest part of the ODM is still pending (persistence), but HTTP binding, query builder and object hydration are working like a charm, and the few bugs that we face in real-world scenarios are solved in a matter of minutes.

And now?

All the namespaces have been changed, so the old Congow\Orient has been replaced by Doctrine\ODM\OrientDB and Doctrine\OrientDB: if you were already using the library, you will need to work on the migration a bit.

You may also want to have a look at the new Packagist page, as it contains the references to the new repository.

I recently found out that the videos recorded at the conference finally made it to Vimeo: enjoy some tarzan english!

Alessandro Nadalin, David Funaro: Graph databases and PHP: time for serious stuff from PLUG on Vimeo.