Solution to token errors

[Ruud]

I'd like to discuss how the token errors problem could be handled. I'm calling it a problem, but that is my view of the way the new token system handles certain things.

The problem
To me it is a problem every time an error is shown to a visitor of a website. Visitors do not always understand something went wrong or do not know how to solve it. In some cases an error message can not be avoided, but in those cases the message must be as clear as possible so even the least capable visitor is likely to understand and fix what went wrong. I should not desire more of my website visitors.

Contao 2.10 introduced tokens that can be used only once. Additionally they are stored in the session; if the session is destroyed all open forms contain invalid tokens.

I'm having problems with the "Invalid request token" page that Contao will show whenever it cannot validate a token. This could be because the token was used, or simply removed when the session terminated. The page can be customized easily, so I will not focus on the technical text it shows which even "technically-above-avarage visitors" do not understand. The page invalid token request page stops all normal processes and does not allow to recover easily; going back will only work if the erroneous form is reloaded with a new request token which will clear all earlier input values the user might have filled in. This is still ok for a simple address form, but many forms contain a textarea which might have contained a lot of text. (I'm typing a long text right now; what if something went wrong and I lost it...)

How to get the problem
In Contao 2.10 I get the problem:

• at any moment I go back to any form that I submitted already and submit it,[/*:m:31w5xi0n]
• whenever I submit a form in an opened tab when I logged out of the website in another tab, just before,[/*:m:31w5xi0n]
• whenever I press the submit button twice (whilst waiting for the first time to load) (ok, can be solved with js),[/*:m:31w5xi0n]
• at times I was not paying attention to what I did; reason unknown.[/*:m:31w5xi0n]

Solutions?
I've already submitted this as an issue which got marked as invalid, so that would mean Leo does not agree this is a problem or misunderstood what I think is a problem. I'd like to submit it again, but that would be useless if I do not explain myself better or offer a better solution then the way Contao 2.10 functions.

Database tokens
I think the problem would be solved if the tokens are added to the database, containing the token and a status (something like open, failed, success, custom_status?). Instead of showing the "Invalid token request" page Contao would be able to determine if the form was processed correctly and show an appropriate message on the form itself on the page where the visitor was before. The tokens can be removed if they go stale or have been used x seconds ago.

The downside is slight increase of database calls, but only when a form was submitted.

So what do you think about the token system and the way the errors are handled? I think I made my views clear!
Also, I already know some people are having problems with it, Thyon's state redirect problem I have not encountered yet.

[ernestmcd]

I'd like to discuss how the token errors problem could be handled. I'm calling it a problem, but that is my view of the way the new token system handles certain things.

This is more than a "problem", it is is a major defect in Contao 2.10.0. I submitted this as an issue to Leo also and was blown off as well with an "Invalid" determination.

I've turned the tokens off for the time being in Contao 2.10.0 installs. Whatever benefits are offered in terms of security are completely negated by the behavior of this token system which simply locks everything up if there is so much as a "hiccup" with a Contao form. If I didn't turn it off, I could spend hours each day just answering phone calls from clients who have a locked up Contao install due to the unsatisfactory way the token system is implemented in Contao 2.10.0.

The only good news about the tokens is that Leo included a way to turn it "off". Obviously the hard working development team did that for a reason, probably because the token system was causing problems for the beta testers as well.

Thank you for the detailed explanation of the behavior of the problem, and your attempt to offer some thoughts about a possible fix.

Ernest McDermon
Snellville, GA

[Medianomaly]

I just want to say thank you for being passionate enough about this to keep fighting your case. I've been following the conversation and agree with everything you've been saying. I've been holding off upgrading/switching until I hear more.

Is this also being discussed on the German forums? Is it perceived as an issue there as well?

I suspect we're not taken as seriously since the English community is very much fledgeling while the German community is much larger, active/proactive, and more vocal. A couple months ago I filed a ticket for a bug. It was almost immediately rejected, and it wasn't until a couple of the other "more respected" members jumped in to back me up that it was taken seriously and finally fixed.

I suspect that's what's going on here, and it will take some people in the "core" community for this to really build some traction.

[ernestmcd]

LOL - I think you're right about the "Deutschland uber alles" attitude. I was stationed in Frankfurt, Germany for three years back during the Cold War era '77 - '80. I love the country, the food, the beer and the friends I met there.

[Medianomaly]

K -- not meant to be any sort of ethnic comment, though.

It's just a fact that the German community (forums, groups, partners, etc.) is much more active -- since it's where the project is based, that makes perfect sense. Could just as easily have been Swaziland.

The English community is much smaller and less active.

So my point had more to do with being more dismissive of the smaller, newer community that they are with the larger, older, more active one. So if the people in that community begin to make noise, it's more likely they will be heard.

[Ruud]

Ok, let's try to stay ontopic indeed! (Although I'm usually not very good at that, and I live next to the German border; no need to upset my neighbors)

I'm sure it is in a lesser manner about being heard; if enough people complain (or the right ones) things will always change. But I think at this moment all we need to do is make a clear description of why this functionality does not work for us (I've seen a point about this increasing customer complaints/queries, which I have not yet had because I disabled tokens). If we have a clear point I see no reason why Leo should call it invalid.

So lets formulate a good reason and if we can a theoretical solution and let's just propose that.

I'll look through the German board tomorrow and see if it is discussed there as well. I know of at least one important German speaking Contao user who does not entirely agree with the way it works, and could always try and see if he wants to chip in on problem/solution.

[winanscreative]

Yeah, I have to agree with the responses here. Not a good way to handle the situation and it is in fact a major defect, as I would not consider that a security feature.

Looking at the ticket it seems Andreas has offered some suggestions that rather trigger an error on the existing page/form, which I think is much more elegant. And the problem still remains of how to handle destroying the session, but it is clear we will need some sort of cache.

The REQUEST_TOKEN idea is a good one, but the implementation is flawed. Hopefully Leo will make a fix.

[ernestmcd]

So lets formulate a good reason and if we can a theoretical solution and let's just propose that.

How about: "Is the token system really a "feature" if we have to disable it out of the box so our clients won't get angry about their new website's behavior?"

[winanscreative]

I think the idea behind the simple token referrer system is actually a great one and is definitely a huge feature.

It's the implementation that I believe is flawed. Interestingly enough, I don't think anyone on the German forum has noticed this being an issue besides Andreas.

Having a form that cannot submit without having valid access from the CMS is a good thing. Generating an error when someone pushes the back button on their browser and resubmits is not. It is a major UI flaw that needs to be addressed. I like Ruud's suggestion above, personally. There needs to be a way to either cache the token in the DB or else figure out a reasonable and secure way to save it to the session and not destroy it each time.

[Ruud]

I don't have more time today to check this. But I tried Drupal, Typo3 and Joomla which all three allowed me to resubmit the contact forms without any message whatsoever.

Having a stateful token like I suggested would provide some benefits over what the other CMS's do while still providing the security that tokens provide.

So, I see we have several votes in favor of changing the way the handling goes, but the German board seems not concerned. Perhaps I should have my starting post translated to German and start the discussion there as well? Should I change parts of it before translating it?

[winanscreative]

I think the way to approach this is to get a push going to recognize the ticket as valid first. Seems to me that is the biggest challenge here. Then we can work through the best way to find a solution.

I would encourage devs to sign up in the Contao ticket system and keep the ticket bumped up. I know Leo runs a pretty tight ship but I think if enough devs get on board for a fix that he is reasonable in the long run.

Here's the link again: http://dev.contao.org/issues/3214

[sal]

I agree that even advanced end users will have troubles handling this error, because this is not something like "Wrong password"
Even if they will take their time to read it once or twice.

I had trouble when I saw it first time at contao translation system until noticed that my session has expired.

And agree with all of you that submiting a ticket is sometimes a pain. Also to get an answer from development team like just for asking where some accepted feature is planned to be included in milestone.

I think it depends if you are a doner and/or how big you are so for next time we should put some $$/€€ before starting conversation or submit a ticket

back to topic:
sort of error handling with proper simple short info for end user explaining what to do would be nice
but as far as I understood there is too many unpredictable situations where this can appear ..so maybe this is the answer why they came up with this sort of "universal error handle"

[Ruud]

Leo's only reply so far has been: "It is supposed to work that way". Not once did he directly address the issue we are having and he even said:

...
I don't know what could be "fixed" here.

Doesn't he read our responses and just say something? It is that or he honestly does not understand the issue or feels the normal website visitor will have no problem understanding the token blahblah he wrote down there... I understand what it says, but I bet many visitors here won't even get that, and they are supposed to work with Contao.

[sal]

yes maybe we are not clear enough and it's our fault ..so I will put here just a little reminder quote from a random google results site:

What is UX?

UX is an acronym for "user experience." It's a term used to describe the overall experience and satisfaction a user has when using a product or system.

Good UX makes people happy and makes businesses more successful. But if UX is ignored, the results can be very bad indeed.

The reservoir of goodwill
No one goes to a site saying, "I will hate this site." Most folks want to like it; they are willing to give you the benefit of the doubt if you make minor mistakes.

But each time something goes wrong, some of their goodwill is lost. Do enough things wrong and your visitors will run out of goodwill and go somewhere else.

Worse still, they don't just leave, they leave disliking your site. And by extension, that means they don't like you, your products, and your services.

Worst of all, studies show that people who have a negative experience are likely to tell 17 other people. If they have a good experience, they will tell 3 - which means a good experience must be as good as it possibly can be!

Good experiences refill the tank
Happily, when you do something right, you start to refill the goodwill resevoir. You build trust in your company, empathy with your products, and loyalty to your brand.

Noted author Steve Krug first described the reservoir concept in his best-selling book, Don't Make Me Think.

[sal]

I will say just one more thing..
Leo is an extremly good developer and understands SEO quite good but when it comes to UX there is allways a battle.
I think UX is as much important as security and should not be ignored in any way.

this also goes for my comments on feature ticket 2670

[winanscreative]

I will also say that Leo is a great developer, and that, yes, sometimes it is tough to consider UX over security and other features. It's also probably really tough to manage a ticket system like Contao has, even with a team of other devs helping.

I think if we give the issue some attention, there will eventually be a resolution that improves the system, and probably adds some whole other additional functionality with it. Contao has been changing so rapidly, and for the better, that I don't think a solution is far off. Just keep raising the issue in a polite and consistnent manner, and I think you'll find that people are very receptive.