Add support for targets::tar_source() and sourceDir()

[lionel-]

Addresses posit-dev/positron#14790
Progress towards #1338

Source effects can now target a directory instead of a single file, which covers the two idioms for loading a folder of scripts: targets::tar_source() and the sourceDir() example from ?source.

# _targets.R
library(targets)
tar_source()                 # defaults to files = "R"
tar_target(x, my_helper())   # goto-def lands in R/my_helper.R

The resolver expands the directory into one target per script, in list.files() order. Load order within the call is respected: code/b.R sees code/a.R but not the other way around. Combined with the previous PR this also means the sourced files see each other, through the sourcing file's scope, without relying on the R/ auto-collation of the first PR.

The PR also adds ad hoc support for sourceDir() with a total hack: any calls to sourceDir() with a resolvable string causes a Source effect. This is to handle the documented pattern in ?source that people are copying around: https://github.com/search?q=sourceDir+language%3AR&type=code. Ideally we'd infer the effects from the list.files() and source() calls. One day.

Positron Release Notes

New Features

• Added support for directory sourcing with targets::tar_source() and a sourceDir() call as documented in ?source (go to definition returns no definition found positron#14790).

Bug Fixes

• N/A

[lionel-]

QA guide from Claude:

---

Covers oak-source/1-r-collation, oak-source/2-source-inheritance and oak-source/3-source-dir.

Setup

Section 5 covers experimental diagnostics, off by default. Turn them on up front, in settings.json:

{
  "oak.diagnostics.experimental.enable": true
}

Everything here needs a non-package project. Open a plain folder with no DESCRIPTION. If one is present the package paths take over and none of this applies.

1. R/ directory collation

A non-package project with an R/ directory gets the same treatment as a package R/ with no Collate:: files load in basename order and each sees the ones before it.

ws/
  R/
    a.R      # a_val <- 1
    b.R      # b_val <- 2

In b.R, a_val resolves and goto-def lands in a.R. Find-references on a_val from either file finds both sites.

Order matters at the top level. In a.R, a top-level b_val should not resolve — b.R hasn't loaded yet:

# R/a.R
b_val            # unresolved: b.R loads after this file
f <- function() b_val   # resolves: the body runs after everything has loaded

That eager/lazy split is the same one as elsewhere. A cursor at file scope sees only predecessors; a cursor in a function body sees the whole directory. local() counts as eager, so a cursor inside a local({ ... }) block at file scope still only sees predecessors.

No cross-collation between directories. Two separate R/ directories (say ws/R/ and ws/sub/R/) don't see each other.

The full search path is still there. This is the regression to watch for, since package code gets the rest from NAMESPACE:

# R/a.R
median(1:10)     # `median` must still resolve (stats, on the default path)

A package R/ file left out of Collate: should keep resolving as a standalone script, not pick up directory collation.

Just-opened files. Open an R/ file in a fresh window and use it before the scan settles. It should behave as if it's in the collation from the start — in particular "unknown symbol" diagnostics shouldn't appear after indexing finishes, which is what a naive implementation does.

2. Sourced files inherit the sourcing file's context

A file loaded by source() sees what the sourcing file had at the call.

# main.R
library(dplyr)
config <- 1
source("helpers.R")
result <- 2

# helpers.R
config           # resolves, goto-def lands in main.R
filter           # resolves to dplyr, inherited from main.R's library() call
f <- function() result   # resolves: runs later, after main.R finished
result           # top level: does NOT resolve, main.R hadn't run that line yet

That last pair is the point of the offset narrowing. The top-level result is a genuine runtime error and should read as unresolved; the one in f's body should not.

Transitive. main.R -> setup.R -> helpers.R should carry main.R's names all the way down.

Several sourcing files union. Each sourcing file is an alternative, not a priority order, so goto-def offers one target per file that binds the name:

# file1.R          # file2.R           # foo.R
x <- 1             x <- 1              list(x, y)
source("foo.R")    y <- 2
                   source("foo.R")

Goto-def on x in foo.R gives two targets, file1.R and file2.R. On y it gives one. Note y is really only bound on one of the two paths, and we report it as certainly bound — the certainty join isn't built (see gaps).

Package and testthat files opt out. A data-raw/build.R that sources a package's R/foo.R must not cost foo.R its NAMESPACE imports, collation siblings, or base. Check that foo.R still resolves everything it did before the source() call existed.

Editing the sourcing file shouldn't churn. Type in main.R somewhere away from the source() call — adding a line at the top, editing a function body. helpers.R shouldn't flicker or lose resolution.

3. Directory sourcing

3.1 sourceDir()

The worked example from ?source. The implementation is a temporary stopgap and recognizes sourceDir() by name alone. Ideally we'd infer the effect from the code:

# main.R
sourceDir <- function(path, trace = TRUE, ...) {
  for (nm in list.files(path, pattern = "[.][RrSsQq]$")) {
    source(file.path(path, nm), ...)
  }
}
sourceDir("code")

helper_from_code_dir()   # should resolve, goto-def into code/*.R

The local sourceDir <- function(...) above the call is exactly what would normally suppress the effect, so if that resolves, the bypass is working.

Flip side of matching on the name: a sourceDir() that does something unrelated will be misread as sourcing a directory. Expected, and the reason the list has one entry.

3.2 targets::tar_source()

# _targets.R
library(targets)
tar_source()          # defaults to files = "R"

tar_target(x, my_helper())   # `my_helper` from R/*.R should resolve

Try all of: bare tar_source(), targets::tar_source(), and a positional tar_source("code"). All three should work.

tar_source("R/utils.R") naming a script rather than a directory should also work — files takes both.

tar_source(files = "code") is not recognized. Deliberate: the path is matched positionally, and the alternative was silently defaulting to "R" and sourcing a directory the call explicitly overrode. A miss, not a wrong answer. Same for tar_source(c("R", "utils")).

3.3 Both

Subdirectories are included (R/models/fit.R counts), matching tar_source()'s recursion. sourceDir() isn't recursive in R, so it gets more than it would really load — the over-approximating direction.

The directory's files see each other, whatever the directory is called — this doesn't lean on section 1's collation, so code/ works as well as R/.

Load order is respected, so it's worth checking both directions with a top-level cursor. Given sourceDir("code") loading a.R then b.R:

# code/a.R
b_val                    # does NOT resolve: b.R hasn't run yet
f <- function() b_val    # resolves: bodies run after the whole call

# code/b.R
a_val                    # resolves: a.R ran first

The middle case is the interesting one to poke at. The sourcing file's own state is snapshotted once, before any target runs, so nothing there can tell a.R from b.R — the ordering comes from the call's target list instead.

4. What this fixes in the issues

Worth checking the reported cases directly.

#15144. Clone https://github.com/benzipperer/ctrl_click. The three ctrl+click targets in main.R: greet_local and greet_static_helper already worked. greet_dynamic_helper is loaded by lapply(list.files("R"), source) and still won't resolve — that idiom needs dataflow we don't have yet. Swap the line for sourceDir("R") or tar_source() and it should.

What #15144 does get: the files inside R/ now resolve each other.

#14790. Functions spread across files directly in the workspace root with no R/ directory and no source() calls. Not fixed by any of this. Confirm it still reports "no definition found" so nobody thinks otherwise.

5. Diagnostics

Both experimental, gated on the setting above.

5.1 source-cycle (Warning)

# a.R
source("b.R")

# b.R
source("a.R")

Both files should report it, anchored at the start of the file (under recovery there's no recorded call to point at). Message says analysis will be incomplete until the cycle is resolved.

5.2 inherited-shadow (Info)

Fires when a name resolves differently because of an inherited layer than it would standalone. Reproducible with no packages involved:

# a.R
source <- identity
base::source("b.R")

# b.R
source("c.R")        # base's `source`, or a.R's `identity`?

# c.R
foo <- 1

base::source on the second line is load-bearing. A bare source("b.R") would be shadowed by a.R's own binding, so the call stops being effectful and the whole edge disappears along with the diagnostic.

The case that matters more in practice is the same shape via an attach: a package on main.R's search path exporting its own library, reaching helpers.R only through inheritance. Harder to set up by hand, but attach ordering is most of what inheritance actually contributes.

Reported in the sourced file, the one being edited, not at the source() call — diagnostics only run for open files, so putting it on the sourcing file would mean the user editing the sourced one sees nothing.

Shouldn't fire when the file's own attach ordering explains the shadow, nor when the sourcing file binds the name only after its source() call.

6. Known gaps

Things to notice and not file.

• lapply(list.files("R"), source) isn't recognized. Needs an abstract-value layer. This is #15144's literal repro.
• A file next to R/ doesn't see R/. In a Shiny app, app.R calling myModuleUI() from R/module.R won't resolve. Shiny's loadSupport() runs inside runApp(), so there's no call to attach an effect to, and we deliberately didn't add a "directory adjacency" rule. Same for a _targets.R that doesn't call tar_source().
• Inheritance replaces collation rather than adding to it. A non-package R/ file that some script also source()s loses its collation siblings.
• No certainty distinction across source sites. A name bound via one sourcing file and not another reads as certainly bound.
• Out-of-workspace source() targets don't resolve at all; nothing tells you why.
• First use may be slow. The first cross-file resolution in a sourced file pays a workspace-wide index sweep. Background priming isn't built. Worth reporting only if it's bad enough to be visible.