fast-glob

Full Changelog: 3.3.0...3.3.1

This release fixes a regression for cases where the ignore option is used with a string (#403, #404).

The public interface of this package does not support a string as the value for the ignore option since 2018 year (release).

So, in the next major release, we will reintroduce method implementations that do not involve strings in the ignore option.

Full Changelog: 3.2.12...3.3.0

Improvements

Method aliases

New methods (glob, globSync, globStream) have been added in addition to the current methods (default import, sync, stream), which eliminate the need to rename the method when importing. In addition, an async alias has been added for the default import, which makes it possible to use this packet with ESM.

Method to convert paths to globs

A new method (convertPathToPattern) has been added in this release to convert a path to a pattern. The primary goal is to enable users to avoid processing Windows paths in each location where this package is used by utilities from third-party packages.

See more details in the pull request.

Bug fixes

• In the past, we mishandled patterns that contained slashes when the baseNameMatch option was enabled, which went against the documented behavior.

• Several problems with matching patterns that contain brace expansion have been resolved. The primary issue solved is when the pattern has duplicate slashes after it is expanded (#394), or the micromatch package does not correctly generate a regular expression (#365).

• All negative patterns will now have the dot option enabled when matching paths. Previously, the !**/* patterns did not exclude hidden files (start with a dot).

• The issue that led to duplicates in the results when overlapping or duplicate patterns were present among the patterns has been fixed. At the moment, we are only talking about leading dot. Other cases are not included. For example, running with the patterns ['./file.md', 'file.md', '*'] will now only include file.md once in the results.

Documentation

A clarifying note has been added for the concurrency option, which provides more detailed information about the Thread Pool utilization.

Full Changelog: 3.2.11...3.2.12

Bug fixes

Fixed an issue introduced in 3.2.7 related to incorrect application of patterns to entries with a trailing slash when the entry is not a directory.

Before changes:

fg.sync('**/!(*.md)')
// ['file.md', 'a/file.md', 'a/file.txt']

After fix:

fg.sync('**/!(*.md)')
// ['a/file.txt']

Thanks @AgentEnder for the issue (#357).

Improvements

This release includes performance improvements for the asynchronous method. For this method we now use an asynchronous directory traversal interface instead of using a streaming interface. This gives up to 15% acceleration for medium and large directories. The result depends a lot on hardware.

You can find the benchmark results for this release in CI here.

Here are a few of measurements on my laptop:

===> Benchmark pattern "*" with 100 launches (regression, async)
===> Max stdev: 7 | Retries: 3 | Options: {}

Name                   Time, ms  Time stdev, %  Memory, MB  Memory stdev, %  Entries  Errors  Retries
---------------------  --------  -------------  ----------  ---------------  -------  ------  -------
fast-glob-current.js   4.390     0.252          6.253       0.015            4        0       1
fast-glob-previous.js  5.653     0.633          6.051       0.056            4        0       1

===> Benchmark pattern "**" with 100 launches (regression, async)
===> Max stdev: 7 | Retries: 3 | Options: {}

Name                   Time, ms  Time stdev, %  Memory, MB  Memory stdev, %  Entries  Errors  Retries
---------------------  --------  -------------  ----------  ---------------  -------  ------  -------
fast-glob-current.js   34.587    1.287          10.654      0.607            11835    0       1
fast-glob-previous.js  41.972    2.086          10.236      1.224            11835    0       1

Full Changelog: 3.2.10...3.2.11

Bug fixes

Yeap, this is another release aimed at fixing problems with detecting brace expansions in patterns. This time, patterns like abc/{a.txt,b.js} was not marked as a dynamic pattern. So, now the regex has been rewritten to a generalized solution as a function to avoid future problems due to the complexity of the regular expression.

Thanks @MurzNN for the report of this problem (#351).

Full Changelog: 3.2.7...3.2.8

Bug fixes

Fix directory matching with trailing slashes (#290)

Thanks @Trott for investigating the problem and the detailed description.

Previously the src/*/ pattern did not work as expected (like src/*).

Double-slash in the middle of the pattern is not collapsed (#330)

Starting from this release, patterns like src//* will work like similar patterns without duplicate slashes. This was done for continuity with other solutions (glob, ls src//*, python, golang, …).

Adjust inefficient regular expressions (#336, #342, #344)

Thanks @Trott for fixing bugs and @XhmikosR for adding the CodeQL action to CI pipeline.

Bug fixes

The previous release (3.2.6) introduced a regression, which makes negative patterns were not applied to patterns outside the current directory.

This release fixes the issue.

Bug fixes

• Now you can use patterns related to the parent directory along with the regular ones. (#310, #316)

// Patterns inside current directory → ['*', './*.js']
// Patterns outside current directory → ['../*', './../*.js']

// Previously you could specify a patterns outside current directory.
fg.sync(['../*.txt']) → ['../file.txt']

// But when the pattern inside current directory was added to them, the behavior broke down.
fg.sync(['*.md', '../*.txt']) → ['file.md'] // The '../file.txt' file exists

// After this fix you can mix both kinds of patterns.
fg.sync(['*.md', '../*.txt']) → ['file.md', '../file.txt']

// Right now we do not support patterns like '{.,..}/*.md'.

Infrastructure

• security

  The glob-parent package has been updated to fix vulnerabilities.

Bug fixes

• Now this packages correctly supports ARM processors (#296, thanks @yozman).

• Fixed behavior that led to Uncaught Error when the package tried to destroy stream multiple times (#282).

  • #53, #58

• Fixed an issue with patterns that starts with the leading slash (#277).

  • Related to patterns with various kinds: /*, /tmp/*, //?/C:/*.

  • #54, #63

:speech_balloon: Common

• An empty pattern now causes an error

:rocket: Improvements

In the #156 issue we've redesigned the deep filter, which controls the reading of directories in depth.

Previously, this filter did not use positive patterns directly (only their maximum depth). The example below shows how many extra directories we read:

{src,fixtures}/**

src → read
fixtures → read
out → read
node_modules → read

Now we apply positive patterns.

{src,fixtures}/**

src → read
fixtures → read
out → skip
node_modules → skip

Synthetic benchmark

More benchmarks can be found here.

{fixtures,out}/{first,second}/*

| | sync, ms | async, ms | stream, ms |
|------- |---------- |----------- |------------ |
| 3.x.x | 13 | 22 | 20 |
| 3.2.0 | 5 | 9 | 8 |

{fixtures,out}/**

| | sync, ms | async, ms | stream, ms |
|------- |---------- |----------- |------------ |
| 3.x.x | 37 | 49 | 52 |
| 3.2.0 | 6 | 10 | 12 |

Real world benchmark

• Globby

• Prettier

• {blocks-*,construct}/**/*.styl (a very large project) 13s → 0.16s

Known issues

• For some cases, there is a noticeable slowdown of 3-6%.

• Patterns containing {a..z} (or similar) may introduce some slowdown.

• Actually, fast-glob is 2 times slower than node-glob in this scenario.

We will work on this in the future.

:tada: Thanks

• @jonschlinkert for the scan method in picomatch that returns parts of the pattern.

• @fisker for early beta feedback.

Bug fixes

Stream is not closed when the receiver is closed (#239)

Previously, we read directories in the stream, even after the receiver is closed. Now we stop reading after closing the receiver by .emit('end'), .destroy() or for await...of.

const fg = require('fast-glob');

(async () => {
    const stream = fg.stream('**');

    for await (const entry of stream) {
        console.log(entry);

        return;
    }
})();

Most likely, in future releases, we will improve integration with streams (#243).

Common

• New method isDynamicPattern as an alternative to glob.hasMagic

• New method escapePath for escaping parts of the paths of the pattern

Bug Fixes

• Matching specific file is not found when pattern contains parentheses

  • ⚠️ Now we route patterns with escape symbol to dynamic patterns

• Match subdirectories starting with . in {dot: false} mode

:gear: Infrastructure

• Move from TSLint to ESLint

• Correct method for the Stream API in the documentation (#217, thanks @bluelovers)

• We have divided the benchmark into two types:

  • The product benchmark is a comparison of performance relative to competitors.

  • The regression benchmark is a comparison of performance relative to the previous version.

• We added a launch of both types of benchmark in CI. Automatically starts only when the master branch is built.

• Now the smoke tests are run for all API's (sync, async, stream).

Bug fixes

• Non-existing directories in the globbing pattern throw an error with the asynchronous API

• The markDirectories option adds extra slashes for every directory in the path with the asynchronous API

The fast-glob3.0.0 was released with one known bug. This release fixes it.

Bug fixes

High memory usage for very big directories (#204)

Highlights

• Entries: 4 000 000

• Before: 4.1GB of RAM (37s)

• After: 0.8GB of RAM (25s)

Explanation

In short, we called 2x replace and startsWith on every entry. Together, that's 12 million calls.

Summary

This release aims to fix architectural issues, increase performance and reduce size of package.

Breaking changes

Since this is a major release, we are introducing a few breaking changes:

• breaking

  Support for the fast-glob@2 is ending.

• breaking

  Require Node.js 8+. But we recommend using 10.10+ for performance issues.

• breaking

  Only forward-slashes in glob expression. Previously, we convert all slashes to the forward-slashes, which did not allow the use of escaping. See pattern syntax section in the README.md file.

• breaking

  Removed options: nobrace, noglobstar, noext, nocase, transform.

• breaking

  Renamed options:

  • extension → extglob

  • followSymlinkedDirectories → followSymbolicLinks

  • case → caseSensitiveMatch

  • brace → braceExpansion

  • matchBase → baseNameMatch

• breaking

  The deep option now accepts only number type and default value now is Infinity instead of true.

• breaking

  The async method was removed. Use fg(/* … */) instead.

• breaking

  The type of returned object when the stats option is enabled is completely changed.

:bug: Bug fixes

• After update from micromatch@3 to micromatch@4:

  • Incorrect matching with curly braces and globstar (#159).

  • Inaccurate comparison within a regular expression (#123, #138).

  • A very long initialization time of filters (#92).

• Now we do not convert slashes in patterns (#173).

• Previously, the baseNameMatch option never worked (#199).

:rocket: Improvements

:speech_balloon: Common

• Package size after installation is decreased: 2.47MB → 0.42MB.

• Package require time decreased: 534ms → 78ms.

:tornado: Speed

Wow! The new version is very fast. At least twice as fast as the previous version. Probably this is the fastest solution in the Node.js world. And that's not all! We will work on performance issues in the future 🐢.

• Speed up between versions for directory with 265k entries: 5x (19s → 4s).

• Speed up between versions for directory with 4kk entries: 4x (4m → 1m).

Look at the benchmarks section in the README.md file.

:gear: Flexibility

Also in this release we have worked on simplifying some scenarios.

Now, thanks to the new mechanism, you can get the type of entry without additional costs! Works only on Node.js 10.10+. Look at the objectMode option.

• concurrency

• fs

• suppressErrors

• throwErrorOnBrokenSymbolicLink

• objectMode

Known issues in this update

• High memory usage for very big directories

Another release of bug fixes

Bug Fixes

Paths not resolved in some cases (#157)

Thanks @stevenvachon for issue reporting :tada:

If the user has passed a . or .. and the absolute option is enabled, the paths of the found entries were not absolute (they contained . or `..).

before

fg.sync('/project/temp/../*.js', { absolute: true }); // → ['/project/temp/../something.js']

after

fg.sync('/project/temp/../*.js', { absolute: true }); // → ['/project/something.js']

The case option not work with static patterns (#172)

Thanks @davidmerfield for issue reporting :tada:

For performance reasons with fast-glob@2.1.0 we introduce static patterns (patterns without glob magic).

Unfortunately, then we forgot about supporting the case (nocase) option. Now the case option works fine with static patterns too. We also improved the documentation for this option.

directory/
  - file.txt
  - File.txt

before

fg.sync('file.txt', { case: false }) // → ['file.txt']

after

fg.sync('file.txt', { case: false }) // → ['file.txt', 'File.txt']

Question mark is not recognized as dynamic glob and fails to find files (#174)

Thanks @vladshcherbin for issue reporting and contributing :tada:

This is also related to static patterns.

Previously we mark patterns like assets/?ss.css to static and tried to find such file on file system. Now it will works fine.

before

fg.sync('assets/?ss.css'); // → []

after

fg.sync('assets/?ss.css'); // → ['asserts/css.css']

Bug Fixes

Broken Stream API when errors occur

Thanks @felixbecker for issue reporting :tada:

#140

This package is able to build tasks on the basis of passed patterns for their parallel execution. In some cases there may be multiple. In the Stream API, each task produces its own Node.js Stream. Once the streams are created, we combine them into a single stream using the merge2 package.

Before this fix, if an error occurs anywhere inside one of streams, it did not propagate to the combined stream, causing an unhandled exception.

For example, when a directory is deleted while it is being globbed with the stream API…

After this fix, all errors will be propagated to the combined stream. One exception is ENOENT errors – they will be ignored in all streams.

A big release of bug fixes

Common

Input values validation (#119)

Unfortunately, not everyone uses TypeScript.
Now we are clearly saying that we only accept strings as input.

Bug Fixes

Absolute negative patterns (#113)

Now we support the absolute negative patterns. You can read more about this in the documentation.

📖 Works only when the absolute option is enabled.

Performance optimizations for non-deep patterns (#120)

We fixed a bug in the mechanism of determining the minimum required depth for reading.

📖 In some cases, we will still read more than we need to, because in the current implementation we cannot accurately determine the depth of the read. But we will work on this further within #53.

For example, the user wrote the following pattern: fixtures/*.

Prior to release of this version, FastGlob could read more directories than it needed:

./fixtures 
└── one 👍
   └── two  👍
      └── three 👍
         └── four ❌
            └── index.js

Right now:

./fixtures 
└── one ❌
   └── two 
      └── three
         └── four
            └── index.js

The markDirectories options does not work when the absolute option is enabled(#121)

Now it works correctly. Thanks @rijnhard for fix it 🎉

The deep options works incorrectly (#129)

Fixed the problem of incorrect depth reading limit when using the deep option.

Bug Fixes

Fix incorrect grouping of negative patterns to tasks (#118)

Thanks @DudaGod for fix it 🎉

Prior to this fix, not all negative patterns could be applied to tasks. For example:

fg('fixtures/first/**/*.md', {ignore: ['fixtures/**/*.md']}).then(console.log);

Give a task:

{ base: 'fixtures/first',
  dynamic: true,
  patterns: [ 'fixtures/first/**/*.md' ],
  positive: [ 'fixtures/first/**/*.md' ],
  negative: [] } // Here must be a negative pattern 

Common

• Now the fast-glob package officially supports Node.js 10.

Flexibility

@nodelib packages

We started migrating to @nodelib packages, which will allow us to fully control the process of package development and some performance improvements (for example, see @nodelib/scandir where we can filter entries before we get fs.Stats) (#104)

Bug Fixes

Fix incorrect search in negative patterns group (#107)

Previously, we skipped negative patterns whose base paths did not fully matched with the base path of the positive pattern. For more details about this situation take a look at #107.

Fix Incorrect negative pattern detection (#101)

In some cases, the ! symbol may not mean that we are working with a negative pattern.

The !(a|b|c) pattern is matches anything except one of the given patterns.

Speed

Now all patterns are divided into two types:

static

The patterns without glob magic.

For example: package.json, /dev/fd, etc.

dynamic

The patterns with glob magic detected by is-glob.

For example: *.json, **/*.js, etc.

This made it possible to reduce the processing time of static patterns. Something like 10-15 times. For more details, see the issue #60. Thanks @pvdlg for reporting.

Bug Fixes

• Fix problem with pattern matching for patterns that contains parentheses (#74, #117).

Flexibility

New options that allow you not to break your brain. User-friendly alternatives for double-negative options:

• nobrace → brace

• noglobstar → globstar

• noext → extension

• nocase → case

⚠️ These options have more weight than their ancestors (no*).

:exclamation: In the next major release, we want to remove the options with double-negation.

Bug Fixes

• Fix problem when onlyFiles & onlyDirectories options is enabled in the one time

Speed

Now we are faster. Almost.

• Now we can excluding from reading directories that apply to negative patterns that ends with /** (slash and globstar) (#45).

This is a maintenance patch release that includes following changes:

• Allow to export Options interface if you are using TypeScript.

• Now we use the @types/readdir-enhanced package instead of your own types.

Some magic for benchmarks 🎉

Please, read 2.0.0 announcement.

Faster. Stable. Flexible. Modern.

:bomb: Breaking Changes

The new version has some fundamental differences.

Drop bashNative mode

In the previous version we used the bash-glob package that provided a wrapper for the ls command to Node.js world. Unfortunately, this package had a lot of problems with shell customizations and macOS. So, in the new version of the fast-glob package we refused to use this package.

Other

• breaking

  Now we only work with array of strings in the ignore option. See more details here: options#ignore.

• breaking

  The onlyDirs option has been renamed to onlyDirectories. See more details here: options#onlyDirectories.

• breaking

  The onlyFiles option is now enabled by default. See more details here: options#onlyFiles.

:rocket: Speed

Now we are faster.

Exclude directories

Now you can say goodbye to your node_modules directory. Please read «How to exclude directory from reading?» section in the documentation.

Smart directory reading

Now we read in depth only those directories that must be read. For example, we have the following pattern:

src/{images,icons}/*/_*.png

In the previous version we read all directories inside the src directory. Now we will read only two directories (images and icons) and one level of directories each of these directories.

src/
├── images/         ← # read
│   ├── file.png
│   └── nested/     ← # read
│       ├── file.png
│       └── wow/    ← # not read
└── icons/          ← # read
    ├── file.png
    └── nested/     ← # read
        └── file.png

More speed optimizations

• Now we use the micromatch as a package for the generation of regular expressions. Why? Because the package has intelligent caching, which slows the search for already prepared the regular expressions in the cache. Something around 15ms for 1000 occurrences.

:traffic_light: Stability

We just became more stable by fixing a few bugs.

:hammer: Flexibility

More options to compare with node-glob

• dot

• followSymlinkedDirectories

• unique

• markDirectories

• absolute

• nobrace

• noglobstar

• noext

• nocase

• matchBase

Other

• Support to ReadableStream adapter.