Topic: FTP uploading issues for Super NES and PC

[JonLeung]

In the past little while, perhaps the last few months, I've been noticing increasing difficulty with uploading files to the Super NES and PC folders.  It often gets stuck and times out, and then I have to wait a while to try again and resume.

Because those two are the folders with the most maps, like 5000 each, I wonder if I'm hitting some sort of limit.  Though it might take until the next day, I do eventually upload everything I intend to upload.  It seems less likely to lag and fail if I don't do lots of files at once, and if I'm not replacing files.  It has more of a problem with the quantity of files, not the sizes of them, hence why the many small Chip's Challenge 2 maps that I am currently uploading are taking a while as I type this.

Any suggestions?  Are there some FTP programs that are better equipped to handle this?

[TerraEsperZ]

Wow, it's been something like 15 years since I've even touched an FTP utility so I won't be of much help. Apparently, there are no technical limits to the number of files in an FTP directory, but there is a limit to the number of files that some programs like Internet Explorer can comfortably display before slowing to a crawl, which tends to be around 5 000 - 10 000 files.

Short of completely overhauling the way the map files are stored and accessed, a quick and dirty way of solving the problem would be to split the directory for a given system in two or more subdirectories, like A-M and N-Z, and doing a mass find/replace to update the link addresses.

Of course, I might also be talking out of my ass since technology has most likely moved quite a bit since my college days.

[Maxim]

Either get a better FTP client, or change the directory structure to avoid the large list, or change technology. I used rsync for some years and it's really helpful if you have a local copy of the site you want to synchronise with. Other times I'd use SSH to do file manipulation after uploading the file somewhere else, e.g. a web upload form.

[Will]

If limitations become too much, perhaps you should divide the Super NES section into two indexes; "Super Famicom" (containing Japanese only releases) and "SNES" (containing US and EU releases). You can also divide the PC index into a "Windows" index and a "DOS" index.

[Katient]

Wow, it's been something like 15 years since I've even touched an FTP utility so I won't be of much help. Apparently, there are no technical limits to the number of files in an FTP directory, but there is a limit to the number of files that some programs like Internet Explorer can comfortably display before slowing to a crawl, which tends to be around 5 000 - 10 000 files.

Short of completely overhauling the way the map files are stored and accessed, a quick and dirty way of solving the problem would be to split the directory for a given system in two or more subdirectories, like A-M and N-Z, and doing a mass find/replace to update the link addresses.

Of course, I might also be talking out of my ass since technology has most likely moved quite a bit since my college days.

i was searching on how to use FTP utility and i am fortunate to discover this thread. i will take note all of this because i believe this infos will really be a big help. hopefully have a positive result. thanks!

[JonLeung]

Long ago I used some version of WS_FTP Pro, but I was annoyed with it creating WS_FTP.log files in every folder.  I don't know if there was a setting or that...

At some point I just used plain old Windows Explorer, which was fine because I'm used to managing files like that anyway.  But that was when I hit those problems where it just failed to upload in the folders I mentioned, the SNES and the PC, but the NES has that problem too.  But I see it's not all of the files, it's just the main HTML file, which can get pretty large in these folders.

I use FileZilla now, and to circumvent that uploading problem, I upload the .htm file last.  When it says it's uploaded 100%, it hangs there for a few seconds.  I can't wait for it to respond because it will say it failed (it didn't) or try to reupload it, in which case I have to wait for it to hit 100% again.  When I see that the number of bytes uploaded is the same as the size of the file, I just cancel it or close the program entirely but it's apparently been uploaded and is fine.  I wish it would just upload without the hang-up though.

So what I don't understand is - and I don't see an option for it - is why it struggles with .htm files over a couple megs, but doesn't have problems with other files of other types that can be much larger?  It's like it's reading the HTML files or something.  Isn't it all data, and any number of things of the same amount of bytes should just upload at the same rate or by the same method?  It shouldn't be FileZilla's, or Windows Explorer's, or any other FTP program's job to do anything with the files but just upload them, leave the reading of them to the browsers when other users are actually accessing them.

I envision it like this, imagine you've got to live somewhere else, so you're ready to move, and you've got all your stuff packed into boxes.  Your friends, or the hired movers, whoever, come along to help bring the boxes from one house to another.  But you've got a bunch of books in some of the boxes, and the movers are having to browse all of your books before they move the boxes of books over.  First of all, that's fricking stupid, you didn't hire/ask them to read your stuff, but to move it.  And then they have no trouble bringing your other boxes, or even your grand piano over, but anytime there's a box of books, nope, gotta browse them all.  And then when there's too many books they get stuck.  Why?  Just move it!

Like, seriously.  I didn't think files were inherently different from each other.  It's just bytes, which in turn are just bits, which are strings of 0s and 1s.  But hey!  Some bytes spell out that this is a header or extension indicating that this is an HTML!  Let's upload it but then choke while reading it!  Gah, that's ridiculous.

I don't get WHY HTMLs have to be handled differently, surely there's got to be an FTP program that doesn't have to snoop.  Unless it's the server's problem?

[Revned]

Make sure you have FileZilla configured to transfer in binary mode. FTP does some character encoding translation for text files unless you turn it off. It hasn't really been necessary for decades, but that's how these things go.

[JonLeung]

Thanks for the tip, Revned.  I'll give that a shot in a few minutes as I upload the Maps Of The Month stuff.

EDIT: Nope, still hangs.  Gragh.