MGET glob support

[oirad]

I encountering some issues when trying to bulk download files matching a specific pattern.

The example of the operation I am trying to do is the following:

ftp> cd test
250 "/test"
ftp> mget *.csv

The error I am receiving is:

{"name":"ftp-srv","hostname":"21e83c34cf28","pid":1,"id":"876b5469-8624-4fd5-85e0-3e795aef2638","directive":"NLST","level":50,"err":{"message":"ENOENT: no such file or directory, stat '/ftp/test/*.csv'","name":"Error","stack":"Error: ENOENT: no such file or directory, stat '/ftp/test/*.csv'","code":"ENOENT"},"msg":"ENOENT: no such file or directory, stat '/ftp/test/*.csv'","time":"2019-10-17T13:26:18.842Z","v":0}

From my understanding this is due to the fact that currently there is no support implemented for this kind of operation and the error is expected.

I noticed that I am able to run the following allowing me to download multiple files, but this results in the system trying to download also sub-folders:

ftp> cd test
250 "/test"
ftp> mget .

In order to try adding this capability the first thing I started looking at is extending the filesystem to add support for this but I'm not sure if I'm missing something, any guidance would be greatly appreciated.

[trs]

Hey, thanks for the issue!

Looks like we need to add support for globbing with the LIST / NLST command.

From a quick search, it looks like when a MGET command is issued, the client just sends multiple GET requests based on the results of the NLST.

Can GET be an alias to RETR?
If not, this command will have to be added to the registry.

Edit: get is just the FTP client's way of sending RETR.

GET is the common user command to download a file instead of the raw command RETR.

So everything should be in place after the globbing support.

[oirad]

Hey @trs thank you for the response.

So probably simply adding the support on the FileSystem will not be enough on my side (without doing something quite hacky).

Looking at the code I see that fs.get is called in LIST on line 19, but then on line 20 is checking if is a directory and returns an array of stats.

17.    return this.connector.waitForConnection()
18.    .tap(() => this.commandSocket.pause())
19.    .then(() => Promise.try(() => this.fs.get(path)))
20.    .then((stat) => stat.isDirectory() ? Promise.try(() => this.fs.list(path)) : [stat])
21.    .then((files) => {
22.        ...
23.    })

So I think that in order to add this, we need to change something on line 20 to see if stat is an array already, then add blob support to the FileSystem on the get method.

Let me know what do you think or if you have better ideas on how this could be implemented.

[oirad]

Hello @trs,

I found a way to get this working by only modifying the filesystem, however I believe the solution I built is not that pretty and works only when the glob is in the file name, not in a folder name.

I would like to hear your thoughts and see if this can be improved both in terms of performances (I'm doing a bit of redundant operations) and in terms of cleanliness.

This kinda works because I trick the section on line 20 of LIST in thinking that is looking at a folder, then filter out the files that do not match in the list method on the filesystem.

const {FileSystem} = require('ftp-srv');
const _ = require('lodash');
const {constants} = require('fs');
const Promise = require('bluebird');
const fsAsync = require('ftp-srv/src/helpers/fs-async');
const nodePath = require('path');
const globToRegExp = require('glob-to-regexp');

class GlobFileSystem extends FileSystem {
  constructor() {
    super(...arguments);
  }

  _cleanGlob(fileName) {
    return nodePath.join(
      nodePath.dirname(fileName),
      nodePath.basename(fileName).indexOf('*') !== -1 ? '' : nodePath.basename(fileName)
    );
  }

  get(fileName) {
    const cleanFileName = this._cleanGlob(fileName);
    const {fsPath} = this._resolvePath(cleanFileName);
    return fsAsync.stat(fsPath)
    .then((stat) => _.set(stat, 'name', cleanFileName));
  }

  list(path = '.') {
    const {fsPath} = this._resolvePath(path);
    return fsAsync.readdir(this._cleanGlob(fsPath))
    .then((fileNames) => {
      const re = globToRegExp(nodePath.basename(fsPath));
      const filteredNames = nodePath.basename(fsPath).indexOf('*') === -1 ? fileNames : fileNames
      .filter((fileName) => re.test(fileName));

      return Promise.map(filteredNames, (fileName) => {
        const filePath = nodePath.join(this._cleanGlob(fsPath), fileName);
        return fsAsync.access(filePath, constants.F_OK)
        .then(() => {
          return fsAsync.stat(filePath)
          .then((stat) => _.set(stat, 'name', fileName));
        })
        .catch(() => null);
      });
    })
    .then(_.compact);
  }

}

module.exports = GlobFileSystem;

Thank you again for the library and hope to be building something that might be useful in a future use-case for this library.