Stabilize #[my_macro] mod foo; (part of proc_macro_hygiene)

[TimNN]

View all comments

Stabilization report

Summary

Allow mod foo; (an "outlined" module) in the input of attribute and derive (proc) macros.

Tracking:

• Tracking issue for procedural macros and "hygiene 2.0" #54727
  • The proc_macro_hygiene feature covers multiple related features. This PR proposes only a partial stabilization

Reference PRs:

• Document modules in attribute & derive macro input reference#2284

What is stabilized

Describe each behavior being stabilized and give a short example of code that will now be accepted.

Currently, mod foo; (an outlined / non-inline) module is explicitly rejected in the input of derive and attribute (proc) macros.

This PR removes the check, allowing mod foo;.

The module is provided to the macro as it is written in the source code, that is in its "not-yet-loaded" form without a body.

// The following was previously rejected and will now compile.

#[my_proc_macro]
mod foo;

#[my_proc_macro]
fn bar() {
    #[path = "foo.rs"]
    mod foo;
}

#[derive(MyTrait)]
struct Foo([u8; {
    #[path = "foo.rs"]
    mod foo;
    0
}]);

What isn't stabilized

Describe any parts of the feature not being stabilized. Talk about what we might want to do later and what doors are being left open for that. If what we're not stabilizing might lead to surprises for users, talk about that in particular.

The remaining parts of proc_macro_hygiene are not stabilized (specifically, attribute macros in places where they aren't currently allowed).

Design

Reference

What updates are needed to the Reference? Link to each PR. If the Reference is missing content needed for describing this feature, discuss that.

rust-lang/reference#2284

RFC history

What RFCs have been accepted for this feature?

This feature has not been explicitly specified by any RFC.

• Ahead of the proc_macro stabilization, #[my_macro] mod foo; specifically was feature gated.
• As a side effect of Track whether module declarations are inline (fixes #12590) #52319, the (unstable) behavior changed:
  • Before that PR, mod foo; was actually provided to macros as the loaded mod foo { ... }
  • After that PR, mod foo; was provided to macros as-is.
• In 2019 (after the proc_macro stabilization, expand: Feature gate out-of-line modules in proc macro input #66078 made mod foo anywhere in the input to derive or attribute proc macros unstable.
  • Before that PR, only proc macros directly applied to mod foo; were unstable.

Answers to unresolved questions

What questions were left unresolved by the RFC? How have they been answered? Link to any relevant lang decisions.

N/A

Post-RFC changes

What other user-visible changes have occurred since the RFC was accepted? Describe both changes that the lang team accepted (and link to those decisions) as well as changes that are being presented to the team for the first time in this stabilization report.

See the RFC history above.

Key points

What decisions have been most difficult and what behaviors to be stabilized have proved most contentious? Summarize the major arguments on all sides and link to earlier documents and discussions.

The main design decision is whether outlined modules should be passed to proc macros with or without their body.

Arguments in favor of "without their body":

• (To the best of my knowledge) the current compiler architecture makes "without their body" the natural choice.
  • To quote @petrochenkov, "it's clear that the current behavior of #[my_macro] mod foo; (the "not-yet-loaded" version of the module is passed to the macro) is natural to the current setup, stable and is not going to change".
• At least I would be somewhat surprised to see a macro (which is operating on the AST) able to see or affect code outside the current file. I would assume by default that they get as input exactly what I have written.
• Function-like procedural macros operate on arbitrary token trees. mod foo; in their input is not even recognized by the compiler and thus not expanded. Attribute and derive proc macros also receiving the unexpanded mod foo; in their input makes the two consistent.
• Other syntactic constructs, e.g. other macros like inlcude!("foo.rs"), are also not expanded before being passed to proc macros. Providing modules without their body would be consistent with that.
• Providing mod foo; to proc macros with its body would (as far as I know) be the only case where a proc macro would not get exactly what is written in the source code as its input.

Arguments in favor of "with their body":

• #[my_macro] mod foo; and #[my_macro] mod foo {} would have different behavior, which means that in the presence of proc macros, migrating an inline module to an outlined one would no longer be a purely syntactic change, but could affect the behavior of the proc macro.
• #[my_macro] mod foo; and #![my_macro] in foo.rs (currently unstable) would have different behavior.
  • Other attributes already have different behaviors depending on whether they are applied as inner vs outer attributes. For example #[cfg(false)] on a module as an outer attribute means the corresponding file can be missig or have syntax errors, while using it as an inner attribute requires the file to be syntactically valid.
• Some macros may need access to (or change) the module body to function properly.

Nightly extensions

Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those?

No.

Doors closed

What doors does this stabilization close for later changes to the language? E.g., does this stabilization make any other RFCs, lang experiments, or known in-flight proposals more difficult or impossible to do later?

None.

If, in the future, there is a desire to make the module body of outlined modules available to proc macros, there are multiple ways to add that in a backward-compatible way (e.g. an explicit opt-in during the proc macro registration, or an API in the proc_macro crate).

Feedback

Call for testing

Has a "call for testing" been done? If so, what feedback was received?

No.

Nightly use

Do any known nightly users use this feature? Counting instances of #![feature(FEATURE_NAME)] on GitHub with grep might be informative.

Searching Github produces 4.4k files. Reviewing the first page of results shows many repositories that haven't been updated in multiple years, so it is unclear which part of the proc_macro_hygiene feature they actually use (other parts of the feature were previously stabilized).

Implementation

Major parts

Summarize the major parts of the implementation and provide links into the code and to relevant PRs.

The restriction is currently implemented as an explicit visitor of the proc macro input, rejecting outlined modules. The visitor is removed in this PR.

Coverage

Summarize the test coverage of this feature.

See the test changes attached to this PR. The added tests explicitly verify that proc macros receive mod foo; as an input without its body, to ensure that the behavior doesn't unintentionally change in the future.

Outstanding bugs

What outstanding bugs involve this feature? List them. Should any block the stabilization? Discuss why or why not.

None.

Outstanding FIXMEs

What FIXMEs are still in the code for that feature and why is it OK to leave them there?

None.

Tool changes

What changes must be made to our other tools to support this feature. Has this work been done? Link to any relevant PRs and issues.

Generally none. The feature is a minor addition to the inputs that (proc) macros accept, and should not require special handling in any of the tools.

rust-analyzer has some existing limitations around handling modules transformed by proc macros (e.g. (on stable) auto-complete does not work if a proc macro transforms mod foo {} to mod foo;). It works fine for simple case where the the proc macro leaves the mod foo; mostly unmodified.

Breaking changes

If this stabilization represents a known breaking change, link to the crater report, the analysis of the crater report, and to all PRs we've made to ecosystem projects affected by this breakage. Discuss any limitations of what we're able to know about or to fix.

N/A: The feature is not expected to cause any breakage.

Type system, opsem

N/A: This feature only affects the AST, not type checking.

Common interactions

Temporaries

Does this feature introduce new expressions that can produce temporaries? What are the scopes of those temporaries?

No.

Drop order

Does this feature raise questions about the order in which we should drop values? Talk about the decisions made here and how they're consistent with our earlier decisions.

No.

Pre-expansion / post-expansion

Does this feature raise questions about what should be accepted pre-expansion (e.g. in code covered by #[cfg(false)]) versus what should be accepted post-expansion? What decisions were made about this?

No. (At least nothing that doesn't equally apply to existing stable proc macros).

Edition hygiene

If this feature is gated on an edition, how do we decide, in the context of the edition hygiene of tokens, whether to accept or reject code. E.g., what token do we use to decide?

N/A: Not gated on an edition.

SemVer implications

Does this feature create any new ways in which library authors must take care to prevent breaking downstreams when making minor-version releases? Describe these. Are these new hazards "major" or "minor" according to RFC 1105?

No.

Exposing other features

Are there any other unstable features whose behavior may be exposed by this feature in any way? What features present the highest risk of that?

No.

History

List issues and PRs that are important for understanding how we got here.

See the RFC History section above.

Acknowledgments

Summarize contributors to the feature by name for recognition and so that those people are notified about the stabilization. Does anyone who worked on this not think it should be stabilized right now? We'd like to hear about that if so.

• @alexcrichton for the work towards the initial proc-macro expansion.
• @petrochenkov for significant work on hygiene and expansion since then.

Open items

List any known items that have not yet been completed and that should be before this is stabilized.

None.

---

r? @petrochenkov for an initial review (feel free to edit the stabilization report if you like)

[rustbot]

stdarch is developed in its own repository. If possible, consider making this change to rust-lang/stdarch instead.

cc @Amanieu, @folkertdev, @sayantn

[TimNN]

stdarch is developed in its own repository. If possible, consider making this change to rust-lang/stdarch instead.

Removed the stdarch changes from this PR.

[TimNN]

@petrochenkov: Based on the labels you added, I assume this looks reasonable to you.

Could you do the cc @rust-lang/lang @rust-lang/lang-advisors step from https://rustc-dev-guide.rust-lang.org/stabilization-guide.html#team-nominations? (I don't think I have permissions).

[petrochenkov]

Ok, if the guide says so, cc @rust-lang/lang @rust-lang/lang-advisors.

[joshtriplett]

@TimNN Did this also gate mod foo; in declarative attribute macros, since that used the same proc macro machinery? (I'd like to make sure attributes have consistent behavior whether they're implemented via proc macros or declarative macros.)

[TimNN]

@joshtriplett: I assume you mean the unstable macro_attr (#143547)?

If so, they also currently reject mod items: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2024&gist=f3f055a47b7c31e54903546c8875caa9

[joshtriplett]

@TimNN Will this PR make them start accepting it? (I'm hoping it does.)

[TimNN]

@joshtriplett: Yes, this PR also allows mod foo in declarative attribute macros.

(I've verified this locally, but this PR also deletes all occurrences of the error message ;).

[joshtriplett]

@rfcbot merge lang

[rust-rfcbot]

@joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[scottmcm]

I think sticking to "you get the tokens, just like everything else with macros" makes sense.

@rfcbot reviewed

(There's an obvious wish to have some kind of post-nameres feature too, but that'd be something different.)

[nikomatsakis]

@rfcbot reviewed

[tmandry]

There's some tension here around whether outer and inner attributes should always behave the same. For macros in particular, we can tell a consistent story that macros are always about tokens.

I can imagine wanting the expansion, but I can also imagine a lot of situations where you don't want that. I think we should go ahead and stabilize the simpler one (semantically and implementation-wise) and defer the other one as future work.

@rfcbot reviewed

[rust-rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[traviscross]

Makes sense to me. We accept different behavior for inner and outer cfg attributes on non-inline modules. The reasons for that apply here too.

@rfcbot reviewed

[rust-rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

[petrochenkov]

@bors r+

[rust-bors]

📋 This PR cannot be approved because it currently has the following labels: S-waiting-on-t-lang, needs-reference-pr.

[petrochenkov]

The reference PR is rust-lang/reference#2284.
@bors r+

[rust-bors]

📌 Commit 5e016b6 has been approved by petrochenkov

It is now in the queue for this repository.

🌲 The tree is currently closed for pull requests below priority 1000. This pull request will be tested once the tree is reopened.