How to asynchronously explore a directory and its sub-directories?

问题

I need to explore a directory and all its sub-directories. I can explore the directory easily with recursion in a synchronous way:

use failure::Error;
use std::fs;
use std::path::Path;

fn main() -> Result<(), Error> {
    visit(Path::new("."))
}

fn visit(path: &Path) -> Result<(), Error> {
    for e in fs::read_dir(path)? {
        let e = e?;
        let path = e.path();
        if path.is_dir() {
            visit(&path)?;
        } else if path.is_file() {
            println!("File: {:?}", path);
        }
    }
    Ok(())
}

When I try to do the same in an asynchronous manner using tokio_fs:

use failure::Error; // 0.1.6
use futures::Future; // 0.1.29
use std::path::PathBuf;
use tokio::{fs, prelude::*}; // 0.1.22

fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
    let task = fs::read_dir(path)
        .flatten_stream()
        .for_each(|entry| {
            println!("{:?}", entry.path());
            let path = entry.path();
            if path.is_dir() {
                let task = visit(entry.path());
                tokio::spawn(task.map_err(drop));
            }
            future::ok(())
        })
        .map_err(Error::from);

    task
}

Playground

I get the following error:

error[E0391]: cycle detected when processing `visit::{{opaque}}#0`
 --> src/lib.rs:6:28
  |
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
note: ...which requires processing `visit`...
 --> src/lib.rs:6:1
  |
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: ...which requires evaluating trait selection obligation `futures::future::map_err::MapErr<impl futures::future::Future, fn(failure::error::Error) {std::mem::drop::<failure::error::Error>}>: std::marker::Send`...
  = note: ...which again requires processing `visit::{{opaque}}#0`, completing the cycle
note: cycle used when checking item types in top-level module
 --> src/lib.rs:1:1
  |
1 | / use failure::Error; // 0.1.6
2 | | use futures::Future; // 0.1.29
3 | | use std::path::PathBuf;
4 | | use tokio::{fs, prelude::*}; // 0.1.22
... |
20| |     task
21| | }
  | |_^

error[E0391]: cycle detected when processing `visit::{{opaque}}#0`
 --> src/lib.rs:6:28
  |
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
note: ...which requires processing `visit`...
 --> src/lib.rs:6:1
  |
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = note: ...which again requires processing `visit::{{opaque}}#0`, completing the cycle
note: cycle used when checking item types in top-level module
 --> src/lib.rs:1:1
  |
1 | / use failure::Error; // 0.1.6
2 | | use futures::Future; // 0.1.29
3 | | use std::path::PathBuf;
4 | | use tokio::{fs, prelude::*}; // 0.1.22
... |
20| |     task
21| | }
  | |_^

What is the correct way of exploring a directory and its sub-directories asynchronously while propagating all the errors?

回答1:

Your code has two errors:

First, a function returning impl Trait cannot currently be recursive, because the actual type returned would depend on itself.

To make your example work, you need to return a sized type. The simple candidate is a trait object, that is, a Box<dyn Future<...>>:

fn visit(path: PathBuf) -> Box<dyn Future<Item = (), Error = Error>> {
    // ...
            let task = visit(entry.path());
            tokio::spawn(task.map_err(drop));
    // ...

    Box::new(task)
}

There is still your second error:

error[E0277]: `dyn futures::future::Future<Item = (), Error = failure::error::Error>` cannot be sent between threads safely
   --> src/lib.rs:14:30
    |
14  |                 tokio::spawn(task.map_err(drop));
    |                              ^^^^^^^^^^^^^^^^^^ `dyn futures::future::Future<Item = (), Error = failure::error::Error>` cannot be sent between threads safely
    | 
   ::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/executor/mod.rs:131:52
    |
131 | where F: Future<Item = (), Error = ()> + 'static + Send
    |                                                    ---- required by this bound in `tokio::executor::spawn`
    |
    = help: the trait `std::marker::Send` is not implemented for `dyn futures::future::Future<Item = (), Error = failure::error::Error>`
    = note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<dyn futures::future::Future<Item = (), Error = failure::error::Error>>`
    = note: required because it appears within the type `std::boxed::Box<dyn futures::future::Future<Item = (), Error = failure::error::Error>>`
    = note: required because it appears within the type `futures::future::map_err::MapErr<std::boxed::Box<dyn futures::future::Future<Item = (), Error = failure::error::Error>>, fn(failure::error::Error) {std::mem::drop::<failure::error::Error>}>`

This means that your trait object is not Send so it cannot be scheduled for execution in another thread using tokio::spawn(). Fortunately, this is easy to fix: just add + Send to your trait object:

fn visit(path: PathBuf) -> Box<dyn Future<Item = (), Error = Error> + Send> {
    //...
}

See the full code in the Playground.

回答2:

I would make several modifications to rodrigo's existing answer:

Return a Stream from the function, allowing the caller to do what they need with a given file entry.
Return an impl Stream instead of a Box<dyn Stream>. This leaves room for more flexibility in implementation. For example, a custom type could be created that uses an internal stack instead of the less-efficient recursive types.
Return io::Error from the function to allow the user to deal with any errors.
Accept a impl Into<PathBuf> to allow a nicer API.
Create an inner hidden implementation function that uses concrete types in its API.

use std::path::PathBuf;
use tokio::{fs, prelude::*}; // 0.1.22
use tokio_fs::DirEntry; // 1.0.6

fn visit(
    path: impl Into<PathBuf>,
) -> impl Stream<Item = DirEntry, Error = std::io::Error> + Send + 'static {
    fn visit_inner(
        path: PathBuf,
    ) -> Box<dyn Stream<Item = DirEntry, Error = std::io::Error> + Send + 'static> {
        Box::new({
            fs::read_dir(path)
                .flatten_stream()
                .map(|entry| {
                    let path = entry.path();
                    if path.is_dir() {
                        // Optionally include `entry` if you want to
                        // include directories in the resulting
                        // stream.
                        visit_inner(path)
                    } else {
                        Box::new(stream::once(Ok(entry)))
                    }
                })
                .flatten()
        })
    }

    visit_inner(path.into())
}

fn main() {
    tokio::run({
        let root_path = std::env::args().nth(1).expect("One argument required");
        let paths = visit(root_path);

        paths
            .then(|entry| {
                match entry {
                    Ok(entry) => println!("visiting {:?}", entry),
                    Err(e) => eprintln!("encountered an error: {}", e),
                };

                Ok(())
            })
            .for_each(|_| Ok(()))
    });
}