redefining struct

[tpapp]

Since structs cannot be redefined, attempting to do so currently gives a failure to evaluate changes warning.

I wonder if this can be worked around by triggering a reload of the entire module in this case.

[timholy]

The problem is that then you have two notions of MyType, MyModuleNew.MyType and MyModuleOld.MyType. You don't see the Old/New when displayed (they both print as MyModule) but in fact they both exist. Any objects::MyModuleOld.MyType that you've created are still around, and you can't use them in methods that dispatch on MyModuleNew.MyType. But since they don't print differently the error messages are extremely confusing, e.g.,

ERROR: MethodError: no method matching foo(::MyModule.MyType)
Closest candidates are:
  foo(::MyModule.MyType) at REPL[2]:1

(The first is really MyModuleOld.MyType and the second is MyModuleNew.MyType.) I don't relish trying to explain that to anyone.

[timholy]

Here's an actual example:

julia> module MyModule
       struct MyType end
       foo(x::MyType) = 1
       end
MyModule

julia> x = MyModule.MyType()
MyModule.MyType()

julia> MyModule.foo(x)
1

julia> module MyModule
       struct MyType end
       foo(x::MyType) = 1
       end
WARNING: replacing module MyModule
MyModule

julia> MyModule.foo(x)
ERROR: MethodError: no method matching foo(::MyModule.MyType)
Closest candidates are:
  foo(::MyModule.MyType) at REPL[4]:3

[tpapp]

I am aware of the issue, but still consider reloading the whole module the best solution. It is recommended in the manual, and is still preferable to restarting Julia, which is the only other workaround I am aware of, and is usually much more time-consuming (especially if one uses Plots.jl etc) than regenerating some values.

I understand your reluctance to deal with this in Revise.jl, as it would be confusing. Perhaps a warning issued in the REPL, or an option that tells Revise.jl to attempt this, would work. But I agree that these are kludges.

[timholy]

I think I wrote that documentation, but turnabout is fair play 😄.

As you probably know, it gets complicated really fast. If you reload A, but have module B that depends on A, you also need to reload B. And (as shown in that manpage) all your variables need to be redefined. Perhaps we could store the entire REPL history for this session and re-execute it? What about statements that, e.g., chop the last 10 bytes off a file, did you really mean to do that twice?

The problems seem quite daunting to me. The "reload all dependent modules" problem may be fairly easily solvable in cases of precompilation because the cache file holds a reliable record of dependencies. (For non-precompiled modules we'd have to rely on a parsing strategy, which might work most of the time but would be fundamentally less reliable.)

The really scary problem is what to do with old variables. I'm not against the idea of trying to do something, but I don't have any good ideas for how to solve it.

[timholy]

OK, I confess you got me thinking: JuliaLang/julia#22721

[vtjnash]

I was wondering: instead of renaming the old type, would it be possible to rename the new one instead? It won't get every case, but perhaps just manually calling eval(rename!(expand())) on everything Revise.jl-aware would cover enough usages to be useful?

[timholy]

That might be work. We'd still have to use methodswith and ask for the callers of the old-type constructor. If the backedges are going away, is that something we can do?

[vtjnash]

It could, but you might also just be able to just look through your parse tree for that symbol. The parse tree will perhaps also be more accurate at finding the actual reference to the type, which doesn't necessarily need to be the same function as the inference tree leading up to an allocation of that type.

[timholy]

Yuck.

julia> struct Foo end

julia> function fakefoo()
           Foo = sqrt
           return Foo(7)
       end
fakefoo (generic function with 1 method)

julia> fakefoo()
2.6457513110645907

Another thing I haven't mentioned yet is that I was planning on trying to scan the eval_user_input args for old types by searching for #RV#, and then warn("I see you have some old types there. It'd be a shame to not use the shiny new ones."). But I could keep a list of specific types to watch out for.

[c42f]

@cstjean just came up with a surprisingly simple but clever idea which could be an effective workaround: https://discourse.julialang.org/t/advice-for-dealing-with-struct-during-development/21732/6

The only obvious downsides being that you'd have to annotate the structs which you anticipate changing before loading the module, and that using such structs would be slow. That seems acceptable to me though.

@timholy, would you consider this (or something similar) for inclusion in Revise? It seems like a developer tool which would be somewhat specific to a Revise-based workflow.

[mauro3]

One thing which that trick doesn't allow is type parameters, or at least no changing type parameters.

[timholy]

I'm interested in exploring this. No bandwidth now, though, not until #243 merges and the associated debugger stack is out there. (And also not a great time to make lots of changes in the Revise codebase, I recommend waiting a week or so.)

[cstjean]

It seems like a developer tool which would be somewhat specific to a Revise-based workflow.

It could be useful for those who include files, too. My inclination would be to make it a separate package, unless there was a specific reason to hook it into Revise.jl. That said, I don't have time to work on it either, so that decision is not up to me.

[adam-r-kowalski]

Could this be resolved by having a system similar to Figwheel. I was an avid user of ClojureScript and this made the development lifecycle amazing. They handled situations like this with their guide for writing reloadable code.

A quick summary is that they had def and defonce (it's a lisp like language). The idea is that you reload everything which is not defonce. This means that if you wish to preserve some state (large dataframes, flux models, etc...) you can tag them with defonce and those will not get reloaded. This way structs can be reloaded and you can safely destroy anything which is not tagged to persist through the reload.

Additionally hooks were provided before-load and after-load to give users a chance to maybe serialize some state down to a file before reloading everything, and then load it back up after.

[timholy]

defonce won't solve the issue here; the problem is not that it tries to redefine a type, it's that it fails to redefine the type, and of course if you never try, you'll fail. Very easy to illustrate:

julia> struct MyStruct
           x::Int
       end

julia> struct MyStruct
           x::Int16
       end
ERROR: invalid redefinition of constant MyStruct
Stacktrace:
 [1] top-level scope at none:0

julia> struct MyStruct
           x::Int
       end

So Julia only complains if you're making changes. Unfortunately if you've made that change in the source code, you'd like it to take effect.

That said, in theory there are places where annotating a block as "load once" might be necessary. An example might be code that initializes a C library, and for which you'd get a crash if you tried to initialize it a second time. I've kind of been amazed that it hasn't really come up yet---either it just doesn't come up, or people aren't reporting problems. I suspect that Julia's precompilation imposes a discipline that makes the large majority of Julia packages well-suited for code-redefinition.

[c42f]

@BeastyBlacksmith mentioned the following on discourse: https://github.com/BeastyBlacksmith/ProtoStructs.jl

[rapus95]

I guess you already considered this, but I'm curious, why you didn't like those somewhat hacky repl tricks:
allow a prefix to be applied to all calls of a given module. Say I want to develop module X, then I'd call set_active_module(X) which would rewrite dosomething() to X.dosomething(), thus, doing the implicit "Main." prefixing, that is said to happen when parsing, automagically redirecting. Alternatively rebind every variable of X into a Main-global variable. That way exchanging bindings on reload would still work. Sure, there's some performance penalty, but since this is an opt-in solution (via the set_active_module(X) function) it could be a welcomed convenience increase.

[timholy]

I'm trying to get away from "somewhat hacky": the main goal in Revise development over the last year or so has been to make it Just Work for almost everybody. I'm fairly pleased with where it is now, and hopefully the slow rate of bug reports recently means that others are too. (If not, please report bugs!) I put problems into different categories, and "should work but doesn't" (aka, a "real bug") is far more serious than "a feature that might be nice but isn't properly supported by Julia itself." By https://github.com/timholy/Revise.jl/issues just about the only "real bug" that I know about is #249. Hopefully most of the infrastructure needed to fix it is already in place, though it's pretty complicated.

Still, it may be getting to the point where it's worth considering tackling this problem again. I had a brief run at JuliaLang/julia#22721 recently, but I'm back to thinking that's hard. Renaming in the other direction might be the the better approach. I'd certainly welcome PRs, though I'm unlikely to accept a solution that only works sometimes.

[vtjnash]

In the context of addressing constant redefinition ala JuliaLang/julia#22721 like JuliaLang/julia#265 (ie with world versioning), we were recently talking about this in the Julialab. It turned out we already have all pieces for it by doing via a small transform: by making a new module that has all the same bindings (but the one) as the old module, we can redefine the type in the new module and then it's defined mechanically if we're referring to the old or new module of that name. But otherwise they have the same bindings (and functions and methods), so using the code won't particularly notice the difference for accessing through the old or new! (Sorry I have no sample code to demo, but happy to answer questions)

[timholy]

by making a new module that has all the same bindings

Same name or different? It's pretty easy to do "different," but then that leaves you with questions to answer about how you integrate the new name into other code.

and then it's defined mechanically if we're referring to the old or new module of that name

Can you elaborate?

[vtjnash]

Same name—since it's a toplevel module, it's just a reference in Base.loaded_modules. Then there's lots of recursive ripple effect where it'll look through all modules with existing references to the old module or name and make a copy of them too. It could be a fair amount to copy, but the end result is a clone of the world with one binding changed (this is almost exactly how the JuliaLang/julia#265 fix works—it clones the "world" rather than modifying and invalidating it).

If you call @eval OldModule <> it'll see the old binding, if you call @eval NewModule <> the new code will see the new binding. The contents of each are the same though, so adding methods via NewModule will make them visible in OldModule too (and vice versa).

[timholy]

Really interesting. If I understand correctly, everything that references that module needs to be recompiled, right? My biggest concern is that could be a lot of recompilation; suppose you change a type in Base, wouldn't that mean that basically everything in your loaded session needs to be recompiled (including Base itself)? Revise currently goes to strenuous effort to reduce the amount of recompilation.

I should say I had a pretty serious stab at this in my teh/typeredef branch and specifically

Revise.jl/src/Revise.jl

Lines 665 to 731 in bd5f1a0

	function revise(queue=revision_queue)
	sleep(0.01) # in case the file system isn't quite done writing out new files
	pdfiles = OrderedDict{Tuple{PkgData,String},ModuleExprsSigs}()
	# Step 1: file parsing
	for (pkgdata, file) in queue
	try
	_, mexsnew = mexs_pair(pkgdata, file)
	pdfiles[(pkgdata, file)] = mexsnew
	catch err
	# parsing errors
	@error "Revise failed to parse $(abspath(file, pkgdata))" exception=(err, trim_toplevel!(catch_backtrace()))
	push!(queue_errors, (pkgdata, file))
	end
	end
	# Step 2: diff computation to determine methods & types to be "deleted"
	methods, types = Set{Method}(), Base.IdSet{Any}()
	for ((pkgdata, file), mexsnew) in pdfiles
	methods_types_to_delete!(methods, types, pkgdata, file, mexsnew)
	end
	# Step 3: traverse all existing methods, looking for ones that are either
	# restricted to one of `types` or specialized for one of `types`
	if !isempty(types) && isdefined(Base, :rename_binding)
	newmethods = Set{Method}()
	action(m::Method) = push!(newmethods, m)
	action(ci::Core.CodeInstance) = ci.max_world = 0
	predicate(obj) = typesmatch(types, obj)
	traverse(predicate, action) # FIXME: set ci.max_world on workers
	# Ensure that we have the expression caches for all new methods
	add_caches!(pdfiles, methods, newmethods)
	# Clear the corresponding definitions
	clear_cache_for_types(newmethods)
	end
	typenames = [Base.unwrap_unionall(T).name.name for T in types]
	# Step 4: invalidate types
	invalidate_types(types)
	# Step 5: delete methods
	delete_methods(methods)
	# Step 6: re-evaluation
	if !isempty(types) && isdefined(Base, :rename_binding)
	# FIXME: this is necessary to get the tests to pass!
	@info "Redefining the following types: $typenames"
	end
	for ((pkgdata, file), mexsnew) in pdfiles
	try
	define_new_methods_types!(pkgdata, file, mexsnew)
	delete!(queue_errors, (pkgdata, file))
	catch err
	# Abort the remainder of the file if we error. This will reduce ridiculously
	# long error dumps if, say, we throw an error upon type definition.
	push!(queue_errors, (pkgdata, file))
	push!(revision_errors, (pkgdata, file, err, catch_backtrace()))
	@error "Revise failed to evaluate $(abspath(file, pkgdata))" exception=(err, trim_toplevel!(bt))
	end
	end
	# Step 7: clean up
	empty!(revision_queue)
	if !isempty(queue_errors)
	io = IOBuffer()
	for (pkgdata, file) in queue_errors
	println(io, " ", abspath(file, pkgdata))
	end
	str = String(take!(io))
	@warn "Due to a previously reported error, the running code does not match saved version for the following files:\n$str"
	end
	tracking_Main_includes[] && queue_includes(Main)
	return nothing
	end

for the high-level overview of how it worked. Of course, since JuliaLang/julia#22721 isn't viable, this branch isn't workable, but you'll see it's quite surgical in what it chooses to invalidate.

[timholy]

@vtjnash, would be interested in restarting this conversation at some point. With Revise 3 out I think most of the remaining brittleness has been eliminated, and it might be time to consider expanding into new territory.

[Amval]

@timholy , hat would be great. For my current Julia use case, I have built a framework. Whenever I am just building client code (on top of the framework), Revise works flawlessly.

The problem is when I am doing work inside the framework. If I need to come close to any struct I am obliged to restart the environment, which is a productivity killer. I love Julia's language design, but unfortunately the current worflow doesn't suit me and my team very well, to the point that we are considering migration :(

I think if there was a workaround this, one of the major paint points of developing in Julia would be solved.

[JinraeKim]

Totally agree with @Amval.
I think that the main drawback of Julia in developing comes from manipulating "the framework" (or, I would say a package).
Using existing packages is very nice though.

@timholy I would like to know whether there is any progress or plan on this issue.

[cstjean]

JuliaLang/julia#40399 is the latest.