Feature request: Base.@public macro for declaring a public name without needing to export it

[DilumAluthge]

Summary

I would like to give package developers a way to declare names that are part of their package's public API without needing to export them.

This feature request has two parts:

1. The Base.@public macro, which declares a name as public without exporting it.
2. The Base.public_not_exported(m::Module)::Vector{Symbol} function, which returns the list of all non-exported public names in the module m.

This feature request is non-breaking. Therefore, it can be implemented in a Julia 1.x release.

Part 1: Base.@public macro

For example, suppose that my package has a public function named fit. Because that word is so common, I don't want to export it from my package. But I want to indicate that it is part of the public API in a structured way that downstream tools (Documenter.jl, JuliaHub, editors/IDEs, etc.) can understand.

So in this example, I could do something like the following:

module Foo

export cool_stuff # public and exported 
@public fit       # public but not exported 

function cool_stuff end
function fit end

function private_stuff end # private 

end # module

Part 2:Base.public_not_exported function

The signature of the function will be: Base.public_not_exported(m::Module)::Vector{Symbol}.

Example usage:

julia> Base.public_not_exported(Foo)
1-element Vector{Symbol}:
 :fit

Related Discussions

There is some related discussion in JuliaDocs/Documenter.jl#1507. However, I really want this to be something that any tool can use, so I don't want it to be specific to Documenter.

[tkf]

Thank you, It'd be indeed nice thing to have!

To fill some details, I think it helps considering nested modules and "export/@public chain." For example, suppose I have

module A
    export B
    @public C

    module B
        export f1
        @public f2
    end

    module C
        export f3
        @public f4
    end

    module D
        export f5
        @public f6
    end
end

I'd consider f1---f4 to be public but not f5 and f6.

Also, it'd be nice if @public f6 can be considered ill-formed so that we can throw a nice error. (It may be a bit tricky to implement this cleanly but explicitly documenting as such let us throw an error in later releases.)

The signature of the function will be: Base.public_not_exported(m::Module)::Vector{Symbol}.

So, given the above point, I think you'd want to return, say, Vector{Tuple{Vararg{Symbol}}}. The element type Tuple{Vararg{Symbol}} is equivalent to what fullname currently returns. Docs.Binding may be another option.

Also, why not something based on optional keyword, like publicnames(::Module; exported = true, recursive = true)?

downstream tools (Documenter.jl, JuliaHub, editors/IDEs, etc.) can understand

It'd be really cool if Registrator.jl can enforce (a part of) semantic versioning like Elm does. Ref: Enforce SemVer like Elm? - Internals & Design - JuliaLang; Also Do people want cargo to enforce SemVer like elm-package does? : rust

[tkf]

An alternative approach to @public macro is to use baremodule

baremodule Foo

export cool_stuff # public and exported
function fit end  # public but not exported
function cool_stuff end

module Internal
import ..Foo: cool_stuff, fit
function private_stuff end # private
end

end # module

and consider Foo.Internal as internal by convention. (Also it's rather clear that using Foo.Internal is using internal even if you don't know about the convention.) I saw this in Haskell and C++: haskell - How, why and when to use the ".Internal" modules pattern? - Stack Overflow

I'm actually using this pattern in most of my recent packages. You can see how it looks like in, e.g., https://github.com/tkf/Reagents.jl/blob/22114bef7331b4ad3d604af5f3255a3fe94f9060/src/Reagents.jl#L1

An upside is that the user can't do using Foo: private_stuff by accident. The downside is that setting it up is a bit tedious and printing of internal types is rather ugly (e.g., stack traces). Also, declaring types this way is rather cumbersome.

Of course, @public approach can be combined to this too.

[mkitti]

As an alternative to creating a new function public_not_exported perhaps there should be also public and exported keyword arguments to names

There is already an imported keyword, so exported is a natural counterpart.

public seems a bit too constrictive if we consider the possibility of other potential access modifiers such as private, internal, protected, or default. Perhaps this keyword should be access or audience?

[tkf]

Another thing to consider may be that @public fails when not all "call signatures" in a generalized sense associated with a name is public. This happens with the name associated with a type:

(1) Some call signatures of the constructor of a type may be considered internal. A strict interface can be implemented with

function _S end
struct S
    ...
    global _S
    _S(...) = new(...)
end
S(...) = _S(...)  # public constructor

but it's rather tedious.

(2) It's not clear if adding type parameters at the end of the parameter list is considered non-breaking (this is the "call signature" of apply_type). If the type is used only at "variant position" (not sure what's the right word) it is non-breaking. That is to say, if S{T}::DataType is changed to S{T1,T2}::DataType,

f(::S{T}) = ...
f(::Tuple{S{T}}) = ...

can survive the update but

struct R
    f::S{T}
end
f(::Pair{S{T}}) = ...

can't.

This, of course, depends on the exact guarantee provided by the API. But it is somewhat unsatisfactory that a basic question like "can this be called?" cannot be answered by looking at the meta data created by @public.

(3) Similar to point (2), changing struct type to abstract type may or may not be considered breaking depending on the exact API.

Possible solution to (1)

The point (1) can be solved by, e.g., extending @public so that

@public struct S
    ...fields...
end

is lowered to

@public S
struct S
    ...fields...
    Base.construct(::Type{S}, fields...) = new(fields...)
end

so that it's easy to expose limited set of call signature from the constructor by defining S(...)= construct(S, ...).

Possible solution to (2) and (3)

A simple-minded solution to (2) and (3) may be to define that:

• The set of type parameters of the non-abstract types marked by @public are also public; i.e., appending type parameters is breaking.
• Appending type parameters to abstract type is non-breaking.
• Changing public abstract type to non-abstract type with compatible type parameters is non-breaking.

[DilumAluthge]

Another thing to consider may be that @public fails when not all "call signatures" in a generalized sense associated with a name is public.

Doesn't export have the same exact limitations? You export a name; you don't export specific methods.

My idea for @public was to be exactly the same as export (except that the name is not actually exported).

Personally, I think that if we have differences in the semantics between export and @public, this will confuse users.

[DilumAluthge]

The counter-argument to my previous comment is that just because export has limitations (that we can't fix in Julia 1.x) doesn't mean that @public` needs to inherit the same limitations.

[tkf]

Right, maybe a more important discussion is "if Base should provide minimal API like @public name"? There are many things we can add to the metadata. For example, I've so far only discussed "call API" but describing "overload API" is another headache. It'd be great if we can encode which argument slot is overloadable by who. That said, just describing public names as mentioned in the OP is already very useful.

[tkf]

I played with the interface suggested in the OP: https://github.com/JuliaExperiments/PublicAPI.jl. It's also possible to add some stricter interfaces like detecting undefined public names and imports of internals.

[adkabo]

With the planned separate-compilation feature, nonpublic names could be excluded from built library artifacts.

[Seelengrab]

My idea for @public was to be exactly the same as export (except that the name is not actually exported).

I understand that to mean "export docs", like declaring that docs & the associated object/function is intended to be used by users of the package. Things that are then either exported or have @public are considered to be "stuff that users of this package are intended to use" and that have to adhere to SemVer. Is that correct?

[Seelengrab]

It would probably be useful to hook @deprecate into this as well, since that is extremely rudimentary at the moment and is probably useful for deprecations in packages as well.

[vtjnash]

Continued in #48819

[DilumAluthge]

This was implemented in #50105