Contao 3 File System

[tsarma]

Implementation of a database-assisted file system, may be a good thing for a site with little files. But as your site grows, It can be a little worrying. Lets say I've a file structure as follows. (this is not far from what is really in my case).

/files
 |_gallery
 :      |_gallery1       (about 400pics)		
 :      |_gallery2       (about 900pics)			
 :      |_gallery3
 :      |_gallery4       (about 150pics)
 :      :
 :      :
 :      |_gallery40
 :      |_gallery41     (about 300pics)
 :      |_gallery42
 :      |
 :      :
 :      :	
 :      |_gallery92      (about 600pics)
 :
 |_downloads            (about 200 files)
 |_members              (about 120 files)
 |_news                (about 3000 files)
 :
 :

As you might have notice each gallery folder contains many pictures and there are lots of them. There are also many other folder containing many more files. Now to use any of these files in Contao 3 Content Element or News, It has to be registered in table tl_files by Synchronizing from Backend.

What do you think now in above case, will the Synchronizing work when I click it?
More often then not I'll bet, your script will timeout or run into memory issues, rendering the website without images!!

People may argue that, if you upload the from Contao's file manager it will work. But then each gallery with more than 100 files! who'll not use FTP for this?
Any files uploaded with FTP, without Synchronizing you can't make a gallery, and if you synch it'll break the site. What a dilemma it is, I think we have to rethink the database-assisted file system in Contao 3.

Database-assisted file system in Contao 3 pro.
- Meta data, very nice I liked it
- Moving files around, OK but who do...

Database-assisted file system in Contao 3 cons.
- As I written mentioned above its a ticking bomb
- without Synchronizing you can't use file from FTP upload
- Exclude folder is not well thought. (If you place an Image in content, later you choice to exclude this images containing folder. Your image is also gone from the FE. Actually this just beats the 2 point from Pro above.)

I hope i am wrong, in my observations, as I would love to see Contao is right. What Do You think!!

Related Topics
viewtopic.php?f=6&t=5978
viewtopic.php?f=6&t=5923
https://github.com/contao/core/issues/5232

Note: If you are not reading this on site https://community.contao.org/. Then you are reading a theft content

[freeyland]

Hi,

I agree with you. I have now a lot of issues after upgrade to contao 3 because of the big amount of files.

[Ruud]

I also see the problem with the dbfs. Luckily I don't have the problem myself, but I can imagine the problems that the system could cause. Below I wrote down whatever ideas I could come up with. It is either something like this or opt to remove the functionality. (Or keep Contao 2 alive indefinitely as the large filecount system)

Leo could make the sync incremental or something; add a sync for large directories which works like the indexer. Detecing directories with too many files would then schedule a sync in a popup/hover or whatever so you can keep on using Contao.

You could mark directories for mass file storage and have those synced by the cron and/or the "large filecount sync".

[tsarma]

The currently sync function in C3 is only good for sites without large number of files. I remember once testing on C3.0.4 on our brand new linux machine. C304 can handel up to 11K with memory_limit of 64MB, execution time 60secs, up to 64K with memory_limit of 128MB, execution time 60secs, and in both cases without any files meta-informations. The idea of dbfs is you can store files meta information like alt, captions etc in DB, with meta information it will be still less.

As Ruud pointed out an incremental sync is a way to go, instead of currently implemented sync function which scans every file and folders every time when you press synchronize button. Sync button should take BE user to folder selection page where the user can explicitly choose which folder to sync. In this way we can safely sync without breaking the system Or be there an option to go conventional way.
Who will think shifting file around, will not result in a broken link in FE unless fixed. I don't think Contao has to play a magic here at the cost of breaking whole system in fews years with ever increasing content files. Another point is files meta informations. Its a nice feature, but again for a website, meta.txt is ok.

I remember Contao working group had a Skype-Conference just about the File system in Contao 3 (on 19th March, from german Contao news). But i don't know what was decided or wether this problem was mentioned at all that time. This dbfs concept needs rethinking and improvement.

[Ruud]

I don't know what they decided in that conference. But the problem with the dbafs is that you either do all files or use a different system. using meta.txt would also create large meta files (if imagenames and such are required) so it has drawbacks to.

Lets find out what they have planned and if it was not discussed ask to look at this specific problem.