10,000 users requires 240mb memory limit

[pbrooks]

Hi all,
We're running a typolight website with 10,000 members, when we had finished the import and attempted to view the members back end nothing loads and not even any errors. I had to increase the PHP memory limit to 240MB to get the members table to load.

The current version of typolight on the system is 2.5.9, have any improvements to optimization been made during recent updates?
You may ask why we are still on an outdated version, this is currently due to some core hacks one developer made to the authentication class. Hacks I am in the process of removing and implementing a similar system to the Open ID module.

[FloB]

Yes, there have been optimizations but I do not know if those have an effect on memory usage. Maybe yes, maybe no. But there will be speed improvements.

I highly recommend you updating the system to TL 2.7 as there are some security and (as mentioned before) speed improvements. Also, because of a new hook, it should be easier to integrate a custom login mechanism.

[pbrooks]

Yes I'll be upgrading asap, certainly glad of the new hooks.
I *think* the current way of thought for login should work. The Open ID method seems pretty sound and I'm experimenting with it now.

I may try to updated install with just the data and see how performance pans out. I'm happy to help solve this issue since 240mb is a damn lot.

[FloB]

Maybe you should also upgrade to PHP 5.3 as it introduces a garbage collector. This might save even more memory.

[pbrooks]

I will look into PHP 5.3 shortly, there's clearly some bottlenecks that need addressing.
Also apologies for missing the other very similar topic.
The situation is the following:
We are currently running the install of typolight in three locations:
1) Production LIVE server VMWare Client (Intel P4 3Ghz 1024 Cache 385648 Memory), Mysql database running on another serve I Don't have access to.
2) My own desktop PC (Intel Core 2 Duo, 2GB Memory)
3) New QEmu virtual machines (Dell PE 1950) with 512MB ram for both Apache & Mysql VMs. I'll be moving the install to this setup after testing/configuration by network provider.

On my desktop if I increase the memory limit to 240 then it loads quickly, however no matter how much I increase the memory limit on the virtual machines there is about a 40 second lag on loading the members table.

What is the bottleneck here to give the 40 second lag? Could it be something within the database?

[FloB]

IMO definitely the DB. Make sure they are on the same network (with a strong connection between them) or even (better) same server. Otherwise you will experience the lag.

[pbrooks]

Hi all,
I'm now debugging this issue, it is not a database communication problem since when the script is running it will consume 100% of the web server CPU.
So far I've tracked the issue down to DC_Table::listView , in my testing we loose:
1 second for the database execution.
3 seconds for fetching the associate array
18 seconds looping through the records.

I am currently looking into the loop and where time is being lost.

[leo]

Have you set the limit filter or are you trying to render a table with all 10,000 members?

[pbrooks]

I'm now trying to render with 15,000 members.
EDIT: And no limit filter, do you mean this is the show filter?

[ramjet]

Hi pbrooks,
the SHOW filter should allow you to view blocks of 100 (or whatever you set in Settings/Back end configuration/Items per page)
I can't imagine why you would want to render all 15000 when you can search/sort/filter and limit to find the ones you want to edit???

[pbrooks]

I wouldn't but users are dumb and are prone to remove all filtering and render the entire lot.

[markhurd]

I wouldn't but users are dumb and are prone to remove all filtering and render the entire lot.

Users are always dumb - it's your job to make sure that doesn't negatively effect their experience with the website.

Could you possibly provide a default filter, in the event all others are removed? Or add pagination so X number of results are retrieved, regardless of filter settings (let the user adjust X but give them options like 25/50/250)? You could also provide a CSV or XLS export option for those few users who really do need a list of every record.

[giel]

I wouldn't but users are dumb and are prone to remove all filtering and render the entire lot.

Users are always dumb - it's your job to make sure that doesn't negatively effect their experience with the website.

I agree but you still have to have the ability to prevent users to do 'dumb' things. Lately, I too ran into this issue for a TL web application, listing records in the backend for a table of over 300.000 (!) records. The users kept on contacting me that their pages failed to load. At first, I could not reproduce the failure, but then I realised that they must have selected 'show all' in the limit filter for that table! And as far as I know, there is no core functionality to disable the 'show all' option from a limit filter in the TL backend...

Maybe a good feature request? What do you guys think?

[markhurd]

I think your idea, or any of the possible solutions I mentioned in my post would all be pretty good additions!

You could make a feature request at http://dev.typolight.org/.

[giel]

I think I'll do that. It's a good thing to make TL friendly to large amounts of data. Also I don't see why Leo would not agree with this. I'll post here as soon as I've added the request.

[stefanjohannsen]

Hi giel

Did you ever post this request? I now have the same problem and can't find a solution :-(

[thyon]

the later version of Contao, sets up a maximum list in the SETTINGS. you cannot view more than those amount of items maximum.