PHP is not Java!

Varnish and saint mode: "no backend connection"

nospam@example.com (Andreas Streichardt) — Wed, 11 Apr 2012 13:33:00 +0200

Recently we implemented some quite sophisticated caching mechanisms using varnish. In addition to just caching, backend errors and invalidated pages had to be handled gracefully whenever possible so we implemented grace mode, saint mode and backend probes.

Everything was working great during development but then we did some real performance testing... The results really gave us some headache because at some point we noticed the following in varnishlog from time to time:

 11 FetchError   c no backend connection

This error appears when (obviously) no backend is available/healthy. The problem: The health checks at that time reported that the backends were alive. Googling didn't help here as this error didn't seem to be one of the standard issues. So i finally found myself digging in the C code and after some while i found out what was going on.

The reason we got this error was a combination of a not-yet completely finished backend and the (poorly documented) saint mode:

We still had quite an amount of 500s returning from the backends because of database inconsistencies and general errors. Now the saint mode maintains a blacklist of URLs per backend. When searching for a backend to handle a request it first checks against this list to see if the URL is blacklisted for specific backends. Varnish will only request a backend that is not blacklisted for the given URL. This is documented.

The undocumented bit: In order to keep the blacklist compact, Varnish saint mode will blacklist a complete backend server silently (!) once a specific number of blacklisted URLs for this backend is exceeded.

If you get frequent 500s from different URLs of all your backends, all will be marked unhealthy over time, resulting in a "no backend connection" error for subsequent requests.

The quick fix was to raise the URL blacklist size (saintmode_threshold). Since this results in longer URL blacklists and thus more memory consumption and longer lookup time, this is not a sustainable solution for production systems. The real solution was to fix all the errors in the beackend.

wetter.com - Relaunch with symfony2, Assetic, Varnish and Twig

nospam@example.com (Andreas Streichardt) — Fri, 16 Mar 2012 11:15:53 +0100

After a lot of hard work, on march 10 a complete rebuild of wetter.com went online. The site is #1 or #2 in germany for weather forecasts since years with millions of visits/day. It was built completely new on PHP 5.4, Symfony 2, MySQL, SolR, Varnish and NginX. The task to re-build the whole site from scratch was given to our partner TFT (Tomorrow Focus Technologies) who invited 100 DAYS into the project to develop the symfony2 parts. This post describes the experiences we made when developing and launching this large scale application.

Continue reading "wetter.com - Relaunch with symfony2, Assetic, Varnish and Twig"

Plat_Forms 2012: Teams needed to represent PHP

nospam@example.com (Gaylord Aulke) — Fri, 17 Feb 2012 11:13:15 +0100

In 2007, the first Plat_Forms contest took place with support of Zend Technologies, University of Berlin, Heise Publishing Company and OSBF. It was a web development platform comparison like it had never been done before: 9 teams in controlled environment doing the same task in a limited time. During that time, the team of Prof. Lutz Prechelt collected data and after the contest, the results together with the data regarding the workflow of the individual teams was evaluated in a scientific way.

Back then, the PHP teams had outperformed Java and Perl in terms of development productivity and usability of the resulting applications. The results of this contests helped to position PHP in many large organisations because they proved common prejudices against PHP wrong: No, PHP is not insecure, no it is not slow, etc.

In 2011 a second contest was helt with 16 teams. Unfortunately the results were not that inspiring this time (as shown here some time ago). Now the third event of this kind is coming up:

Plat_Forms 2012 will take place on April 3rd and 4th, 2012 in Berlin, Germany.

Plat_Forms 2012 will focus on scalability and cloud computing. Unlike in 2007 and 2011, this year's teams will implement a highly scalable web service on Amazon Web Service infrastructure.

Again, multiple teams consisting of 3 persons each are invited for each web programming platform: PHP, Ruby, Perl, Java. They are searching for strong PHP teams with the will to compete.

PHP needs you! More information here.

Plat_Forms 2012 announcement

nospam@example.com (Gaylord Aulke) — Tue, 31 Jan 2012 14:48:35 +0100

Heise and the university of Berlin just announced the next iteration of the Plat_Forms programming contest. This time, the task all teams need to implement will be more "cloudy", not that frontend-heavy. Hopefully, PHP will perform better than in 2011 this time. If you have a team of 3 PHP developers and if you think you can compete with Ruby and Java: registration is open!

Plat_Forms 2011 aftermath

nospam@example.com (Gaylord Aulke) — Thu, 05 Jan 2012 11:11:23 +0100

In January 2011, the second iteration of the Plat_Forms contest had been conducted by FU Berlin Heise and OSBF. 16 Teams with 3 developers each were all given the same task. They had to implement as much as possible of the given requirements in a given time span in a controlled environment. After the event, the resulting code has been evaluated by the team around Prof. Lutz Prechelt and Ulrich Stärk. The results were then presented on November 25, 2011.

For the PHP Side, i was asked to select the participants. To provide a good coverage of current PHP best practices, i selected 2 zend framework teams, 1 symfony team and a flow3 team (as an additional influence besides the general purpose frameworks).

After the great success of PHP in the previous contest in 2007, we expected after all the improvements in the PHP space during the last years, that PHP would be even more successful than before. But we were supprised:

So how can this be interpreted?

Java was almost constant, slightly improved coverage. Perl decreased somewhat, that might be discussed elsewhere. Ruby was very good, but there was no numbers for 2007 so no trend can be derived here.

But what happened to PHP? Less coverage than 2007 and much less consistency between the teams (one quite good, the other 3 rather bad). Did we select the wrong teams? Did they have a bad day?

Why did all the work the PHP community did during the last years in terms of software architecture, frameworks and quality not lead to better productivity?

Ok, the results for robustness have improved compared to 2007 for PHP. But the size of PHP applications was less consistent than in 2007: Some PHP applications were quite compact, others were as big as the Java applications but while covering less functionality.

To me these results match my observations during the last years in PHP space. The idea of PHP being "super-productive" needs to be questioned at least. Maybe the PHP community is on a journey and the destination has not been reached yet. We will see with the results of the next iteration of this contest...

Side Notes:

While Ruby Teams spent much more time writing automated tests than everybody else, they still got the highest functional coverage.

While in 2007 the PHP teams were most interested in customer's wished, in 2011, Ruby asked the most detail questions.

Neues SSL Zertifikat mit Plesk 9.3

nospam@example.com (Gaylord Aulke) — Mon, 19 Dec 2011 15:33:33 +0100

Alle Jahre wieder: Das SSL Zertifkat läuft ab und ein neues muss her. Und das will natürlich installiert werden. Kein Problem: Wir haben ja Plesk. Einfach SSL Zertifikat-Manager aufrufen und hochladen. Schön ist die Welt.

Schade nur, dass Plesk (zumindest die 9.3) dabei zwar das Zertifikat für den Webserver updatet, aber die für Qmail, ImapD und POP vergisst. Also doch wieder Kommandozeile.

Bei der Gelegenheit hab ich mir auch die Apache Config mal angeschaut und muss sagen: In Zukunft wohl immer Kommandozeile...

Mein iPhone und ich

nospam@example.com (Gaylord Aulke) — Mon, 05 Dec 2011 12:34:07 +0100

Ich habe mein iPhone gewaschen. Um genau zu sein, die Reinigung Trieb hat es gewaschen, ich habe es nur in der Jackentasche vergessen beim Abgeben. Der Schaden ist überschaubar: Es was ja nur ein 3GS und ich wollte sowieso ein neues.

Was mich aber wirklich daran aufregt: In den letzten Wochen hatte ich bestimmt 3-4 Alben bei iTunes gekauft, die ich noch nicht auf meinen PC gesynct hatte. Warum auch? Geht doch alles übers iPhone. Und jetzt weiß iTunes zwar, dass ich diese Alben gekauft und bezahlt habe, erneut herunterladen darf ich sie aber nicht. Und das liegt nicht an Apple: Die Apps, die ich einmal gekauft habe, kann ich beliebig oft neu herunterladen. Nur Musik nicht.

Das regt mich auf, weil ich keine Musik illegal lade, sondern sie kaufe. Ich habe sie bezahlt und darf sie trotzdem nicht anhören. Das mag ich nicht akzeptieren. Nicht in 2011. Aber die Musikindustrie - die einzig von Menschen wir mir lebt - interessiert sich nicht dafür was ich will. Sie interessieren sich nicht für ihren Markt. Sie haben nur Angst um ihr ausgedientes Geschäftsmodell...

Die gefilterte Realität

nospam@example.com (Gaylord Aulke) — Tue, 28 Jun 2011 20:37:10 +0200

Es ist Sommer, und da machen die Journalisten sich auf die Suche nach Überbrückungsthemen, die man immer mal bespielen kann ohne sich dabei zu verbrennen und ohne wichtige Dinge vor einer urlaubsgeschwächten Leserschaft zu verbrauchen. Offenbar modern ist daher gerade die bildungsbürgerkompatible Kritik an einem neuen Trend: Informationsbeschaffung aus Sozialen Netzwerken. Es wurde bemerkt, dass viele Leute nicht mehr den Spiegel lesen oder die TAZ oder die Nachrichten um acht schauen. Stattdessen beziehen sie ihre Nachrichten aus sozialen Netzwerken:Irgendwer schnappt etwas auf und twittert oder postet es, mit Quellennachweis. Gefährlich wäre das, wird da gewarnt, weil es eine Filterung der Realität darstelle. Man bekäme nur noch die Informationen, die Menschen interessant finden, mit denen man befreundet ist. Das wäre eine Einschränkung der Wahrnehmung, wird da gewarnt. Tatsächlich? Verödet man durch soziale Netze? Wird die Welt kleiner dadurch, dass man nicht mehr nur von 5 Arbeitskollegen und 3 weiteren Leuten, denen man am Tag begegnet, Informationen aufschnappt, sondern von ca. 200 Facebook-Freunden? Maßt Facebook sich etwa an, die mich interessierenden Informationen sogar vorzufiltern und so alle auszuschließen, für die ich mich in den letzten 5 Monaten schon nicht interessiert habe? Wo doch der Spiegel oder die TAZ jede Nachricht direkt und ordentlich aufbereitet an die werte Leserschaft verbreitet. Da findet natürlich nur eine gute Filterung statt. Mit Journalistischem Hintergrund. Da weiß man ja, was wichtig ist und was nicht. Mich beschleicht ein wenig der Verdacht, dass hier ein Berufsstand um sein Recht auf Informationsfilterung bangt und deshalb fleißig anprangert, was in einigen Jahren ihren Job gefährden wird...

Agile Fix Price

nospam@example.com (Gaylord Aulke) — Mon, 06 Jun 2011 14:37:36 +0200

Working with SCRUM is quite straight forward as long as you are inside the origanization that owns the project. But once you are an external supplier and sell the implementation of a certain piece of software to a customer, the game changes. As a supplier, the perfect scenario would be to be paid by the hour. In this case the customer can supply the product owner and freely define and prioritize requirements for each sprint. The project risk (including the risk of having chosen the wrong supplier) is then completely on the customer side.

This is where the purchase department of large companies kicks in: They want to buy software like a car: "Tell me the features, give me a price, let us lower the price a bit, then you deliver on time, we check the deliverable. If everything is ok, we pay". Since 30 years, software develpment never worked this way, and since almost the same time, people think they just need to try harder next time.

This game is ok for the supplier as long as they put enough buffer into the budget and into the time plan and prevent scope creep during the project. The bid needs to be so high that in the best case the supplier gets something on top of his usual daily rates for taking the risk. In worst case, his margin will go down... In such a project environment you cannot let the customer define prioritization of the Tasks to implement because the project is then not about delivering the best possible solution anymore. It is about fulfilling the contract. The product owner (speaking Scrum again) needs to be inside the suppliers organization. The customer is only a stake holder. The potential of agile is very limited in such an environment.

A solution that appears more suitable for both sides is described in an article by Bernd Österreich (sorry, german) and at InfoQ as "Agile Fix Price". The idea is basically to make a normal offer with normal estimation (broken down to individual requirments), then you make a Special Contract that says:

- We implement all the required functionality (link to Requirments-Spec) for XYZ EUR (or US-$)

- We work with an Agile Method (for example: SCRUM), customer supplies the Product Owner

- In parallel to the agile develpment process, we deploy a bean counter who always checks the implemented functionality against the original requirments. Each deviation is written down in form of a Change Request

- Change Requests can be additional work or less work compared to the original offer. They are estimated in person hours

- Shortly after each planning session, the list of change requests is updated and (very important:) discussed with and approved by the customer

- Once a CR is approved, the customer can decide if the want to extend the budget (and timeline if needed) or if they want to leave out other functionality of comparable effort.

It seems like this method is a good compromise between supplier and customer stakes. The "dark side" of this approach is that the "bean counting" and discussing the detected changes with the customer take a lot of time. This is hard work that nobody wants to do. It is still extremely important to prevent scope creep while keeping as much flexibility as possible in the development process.

Our rules of thumb are that the "beancounter" job takes about 1-1.5 days per 2-weeks sprint. The discussion with the customer is about 45 minutes per sprint.

The beancounter needs that much time because she needs to be able to stand an argument with the customer over each detail in the CR. Otherwise, the customer will be very upset to be confronted with unreflected CRs that turn out to be part of the original requirements during discussion. The needed time depends a lot on the quality of the requirements and on the structure of the calculation documentation.

It is further very important that the customer understands and supports the reasons for this aproach and sees that:

- supplier cannot take the risk of working more if there is no chance of working less than expected

- customer wants to have benefit from agile, so they need to pay the price of additional communication effort with SCRUM

- customer wants to have fix price, so they need to pay for the additional "bean counting" overhead in the project

If this is clear, and if the atmosphere during the meetings is one of a bazaar more than of a cathedral (to quote an old open source article) the agile fix price can be the least annoying of all options to do agile project work...

UMG making friends online

nospam@example.com (Gaylord Aulke) — Wed, 02 Feb 2011 14:26:54 +0100

One of the cool things about the internet is that you cannot stop it. Egypt's government try hard, but they fail. This medium is resisting censorship better than any other communication channel. Videos like this show the movement, how the people of egypt demonstrate against the government. And they cannot prevent this from being published.

However, in germany this video cannot be viewed, because it contains a song of Chris de Burgh. Universal Music obviously think profit is more important than human rights. I still do not understand their online strategy. I mean: Would i buy this song instead of listening to it as background music for this clip? Very unlikely.

Plat_Forms Contest 2011: it was great fun

nospam@example.com (Gaylord Aulke) — Sat, 22 Jan 2011 21:33:17 +0100

On tuesday and wenesday last week, the 2011 version of the Plat_Forms contest took place. It was organized by FU Berlin, OSBF and publishing company Heise.

16 Teams of 4 (+1) different web application development platforms were given the same requirements and two days of time to develop as much as possible. I was again amazed by the great atmosphere and the cooperative attitude of the teams. This year, there were teams of PHP, Perl, Java, Ruby and one Javascript-On-The-Server Team.

The organizers around Prof. Lutz Prechelt and Eduard Heilmayr did a fantastic job to provide a great environment to work in. They will be evaluating the work of the different teams now and present their scientific results in a couple of months from now.

Some impressions of the contest can be seen in the contest blog.

Spot the java guys...

The jury

Adding HTTP Headers to PHP SOAP requests

nospam@example.com (Gaylord Aulke) — Wed, 08 Dec 2010 10:46:12 +0100

I was confronted lately with the requirement to send a SOAP request with additional HTTP headers attached to it (Not SOAP headers, which are passed as part of the SOAP XML envelope). After some research i only found some quite hackisch solutions.

But then Dmitry pointed my to a solution using stream context (which was not too obvious to me):

$x = new SoapClient($wsdl, array("stream_context" =>
    stream_context_create(array("http"=>array(
        "header"=> "Accept-language: en\r\n".
                   "Cookie: foo=bar\r\n"
    )))));

$x->remoteFunction();

Plat_Forms contest 2011

nospam@example.com (Gaylord Aulke) — Fri, 05 Nov 2010 19:40:34 +0100

In 2007 there was a programming competition conducted by the university of berlin and "Heise Verlag":
Different programming platforms for web applications (Java vs. PHP vs. Perl) should be compared and the result were analysed in many aspects. I took part as a member of the Zend team and had a lot of fun there.
Now Prof. Prechelt from FU Berlin is setting up another challenge. In January 2011, the contest will be repeated. With more platforms and even more fun this time. The official announcement can be seen here: http://www.plat-forms.org/platforms-announcement

We are searching for 3-4 teams from the PHP area to participate there.
If you feel you and your 3-person team belong to the top level PHP developers and if you want to take the challenge and compete against Java, .NET, Ruby, Python and Perl teams, feel free to apply now!
See you there

Interesting view on Copyright

nospam@example.com (Gaylord Aulke) — Wed, 11 Aug 2010 08:27:00 +0200

In issue 31, the german magazine "Der Spiegel" has an interesting article about the impact of copyright on the book market in the 19th century. The copyright was originally invented to protect the author. It was said that no author would write anything without his copyright being protected. But it turned out the copyright protected the position of the publishing companies much more than the ones of the author. The books in england (where copyright existed) became a luxury good because the publishing companies sold them for very high prices. In germany, without copyright, books were very cheap and the publishing companies focussed on making printing cheaper (inventing the paperback). During this time, germany produced way more new content than england.

Interesting analogy to today's situation in the music industry, IMHO...

Netzneutralität? Extrapreise für Datentransfer? Ja ist denn schon Weihnachten?

nospam@example.com (Gaylord Aulke) — Thu, 22 Jul 2010 18:32:39 +0200

Seit Jahrzehnten gibt es das Internet jetzt schon. Und in all dieser Zeit war es immer so, dass Dienstanbieter für das Einstellen ihrer Inhalte Geld bezahlen mussten. Und die Konsumenten mussten für den Zugang zum Netz Geld zahlen. Dazwischen werden über Peering-Abkommen Bandbreiten "getauscht". So wurde das weltweite Netz in Rekordzeit aufgebaut und ausgebaut. Inzwischen kann jeder sich einen Zugang und fast jeder sich einen eigenen Server leisten. Ist das nicht großartig?

Aber jetzt wollen große Netzbetreiber von Anbietern bandbreitenintensiver Dienste extra kassieren. Das klingt erstmal als ob es ja nur die "richtigen" trifft. Im Endeffekt sorgt es aber dafür, dass es in Zukunft im Datennetz flächendeckend privilegierte und weniger privilegierte Anbieter geben wird. Das ist heute schon so: Große Anbieter können sich eine weltweite Verteilung ihres Contents via CDNs wie Akamai, Limelight oder CDNetworks leisten (die wiederum Geld an die Netzbetreiber zahlen). Und "dicke" Anbindungen für ihre Rechenzentren in der Nähe von zentralen Knotenpunkten, um den Weg auf die Datenautobahn so vielspurig wie möglich zu halten. Aber wenn die Daten einmal im Backbone angekommen sind, werden alle mit dem gleichen Recht überall hin befördert.

Wie soll das nun in Zukunft aussehen? Wird jeder größere Dienstanbieter Verträge mit jedem wichtigen Backbone-Provider weltweit brauchen? Muss man wie in grauer Vergangenheit für jede Brücke, jedes Königreich am Rande des Weges auf der Datenautobahn, Wegzoll bezahlen? Vielleicht schließen sich die Netzbetreiber zu sowas wie der GEMA zusammen und erheben pauschal Beiträge für jede IP-Adresse? Und wenn z.B. die Telekom bestimmt, welche Pakete wie zu Kunden vermittelt werden, wird sie dann nicht die Pakete ihrer eigenen Inhaltsangebote massiv bevorzugen?

Ich glaube auch die Telekom denkt nicht wirklich, dass man die globale Ökonomie des kabelgebundenen Internets noch umkrempeln kann. Aber im Mobilbereich, da ist noch alles drin. Da kann man abrechnen, da kontrolliert man die Netze. Ich schätze in diesem Bereich wird sich die Auseinandersetzung um Netzneutralität mit anders verteilten Karten abspielen...

UPDATE: Über Isotopp's Blog gefunden: Google's caching and peering strategie. Bei Google hat man die Ökonomie des Internets wohl besser verstanden...